651-700 of 10000 results (45ms)
2018-01-02 §
13:44 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1098:3317 - T174569 (duration: 00m 51s) [production]
13:42 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw1329.*.eqiad.wmnet [production]
13:41 <elukey> enable live traffic for new appservers mw1329->mw1333 (T165519) [production]
13:00 <moritzm> installing openssl updates on remaining mw* hosts in eqiad [production]
12:25 <jynus@tin> Synchronized wmf-config/db-codfw.php: Repool db1055 & db1056 as x1 replicas (duration: 00m 51s) [production]
12:24 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Repool db1055 & db1056 as x1 replicas (2nd try) (duration: 00m 51s) [production]
12:21 <akosiaris> empty ganeti1008 for kernel downgrade. T181121 [production]
12:11 <jynus> add missing mysql grants to db1055 and db1056 [production]
11:42 <moritzm> installing ncurses security updates [production]
11:39 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Revert: Repool db1055 & db1056 as x1 replicas (duration: 00m 51s) [production]
11:32 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Repool db1055 & db1056 as x1 replicas (duration: 00m 50s) [production]
11:31 <mobrovac@tin> Finished deploy [citoid/deploy@ee0bdf4]: Update to service template v0.5.4 - T151394 (duration: 04m 19s) [production]
11:30 <jynus@tin> Synchronized wmf-config/db-codfw.php: Repool db1055 & db1056 as x1 replicas (duration: 00m 50s) [production]
11:27 <mobrovac@tin> Started deploy [citoid/deploy@ee0bdf4]: Update to service template v0.5.4 - T151394 [production]
09:45 <oblivian@puppetmaster1001> conftool action : set/pooled=inactive; selector: cluster=api_appserver,name=mw1(1.*|200).eqiad.wmnet [production]
09:44 <_joe_> setting api_appservers in the mw1180-1200 range to pooled=inactive, T183895 [production]
09:43 <oblivian@puppetmaster1001> conftool action : set/pooled=inactive; selector: cluster=appserver,name=mw1(1.*|200).eqiad.wmnet [production]
09:37 <_joe_> setting appservers in the mw1180-1200 range to pooled=inactive, T183895 [production]
09:28 <godog> reboot ms-be1033 - T183724 [production]
08:52 <_joe_> restarting also mw1226-8, mw1223, mw1201,mw1203, mw1205-7 [production]
08:36 <_joe_> likewise for mw1285,mw1235,mw1232 [production]
08:29 <_joe_> restarting hhvm on mw1280,1282 for the same reasons [production]
08:26 <_joe_> restarting hhvm on mw1317, multiple threads stuck in HPHP::jit::enterTCImpl [production]
08:23 <elukey> restart druid coordinators on druid* to pick up new jvm settings [production]
08:19 <_joe_> restarting hhvm on mw1313, concurrency HPHP::VariableUnserializer::unserializeVariant [production]
08:06 <marostegui> Deploy alter table on db1039 (already depooled) - T174569 [production]
07:56 <marostegui> Deploy schema change on dbstore1001.s7 - T174569 [production]
06:51 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1110 (duration: 00m 52s) [production]
06:42 <marostegui> Stop db1110 and dbstore1002.s5 replication in sync [production]
06:28 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1110 to reimport dewiki.langlinks on dbstore1002 (duration: 00m 50s) [production]
2018-01-01 §
22:45 <eileen1> re-enable silverpop jobs process-control config revision is 4a4c61ae1b [production]
22:36 <eileen1> update process-control to process-control config revision is 65b65972a9 (silverpop password change) [production]
21:01 <addshore@tin> Synchronized php-1.31.0-wmf.12/extensions/WikimediaEvents: [[gerrit:399627|Add onBeforeInitializeWMDECampaign]] T182797 T182794 PT 2/2 (duration: 00m 53s) [production]
20:59 <addshore@tin> Synchronized php-1.31.0-wmf.12/extensions/WikimediaEvents/WikimediaEventsHooks.php: [[gerrit:399627|Add onBeforeInitializeWMDECampaign]] T182797 T182794 PT 1/2 (duration: 00m 55s) [production]
2017-12-30 §
18:22 <ejegg> disabled failing silverpop data fetch jobs [production]
2017-12-29 §
18:31 <mutante> gerrit: restart service to apply change from Dec 20th, avoid logspam due to unapplied config change [production]
15:38 <ariel@tin> Finished deploy [dumps/dumps@51d4fd6]: enable pageslogging dump in parallel jobs (duration: 00m 02s) [production]
15:38 <ariel@tin> Started deploy [dumps/dumps@51d4fd6]: enable pageslogging dump in parallel jobs [production]
14:46 <apergos> restarted puppetdb on nitrogen, puppetdb refusing to hand out facts again [production]
08:34 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1110 (duration: 00m 52s) [production]
08:19 <marostegui> Stop replication in sync on dbstore1002.s5 and db1110 [production]
08:03 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1110 to reimport dewiki.imagelinks on dbstore1002 (duration: 00m 52s) [production]
07:43 <marostegui> Fixing dbstore1002 dewiki.pagelinks [production]
2017-12-28 §
14:18 <marostegui> Power cycle pc2005 as it is down [production]
2017-12-27 §
19:54 <godog> power reset ms-be1033 [production]
19:47 <godog> reboot ms-be1033 - T183724 [production]
17:00 <twentyafterfour> restarting apache on phab1001 [production]
15:54 <mobrovac@tin> Started restart [electron-render/deploy@94d27d7]: Bounce Electron, stuck - T174916 [production]
2017-12-26 §
16:33 <mobrovac@tin> Finished deploy [mathoid/deploy@63b2ddc]: Bring back Mathoid in codfw to v0.6.5 in sync with eqiad - T179419 T172767 (duration: 01m 44s) [production]
16:31 <mobrovac@tin> Started deploy [mathoid/deploy@63b2ddc]: Bring back Mathoid in codfw to v0.6.5 in sync with eqiad - T179419 T172767 [production]