551-600 of 10000 results (31ms)
2016-03-09 §
12:25 <elukey> added rdb1003 back to the Job Runners queue. All the jobchron processes on jobrunners/videoscalers need to be restarted. [production]
12:07 <elukey@tin> Synchronized wmf-config/jobqueue-eqiad.php: Add rdb1003 back to the Redis JobQueue pool after maintenance (duration: 00m 34s) [production]
12:00 <elukey> re-enabled puppet on rdb1004 [production]
11:47 <moritzm> uploaded firmware-nonfree 20151018 to jessie-wikimedia on carbon [production]
11:39 <godog> rolling restart cassandra in staging after merging https://gerrit.wikimedia.org/r/#/c/275917/ [production]
11:27 <jynus> setting up master-master cross-datacenter replication for s2 (db1018-db2017) [production]
11:20 <ema> hhvm restarted on mw1140 [production]
11:04 <jynus> deployed manually new code at dbtree (noc.wikimedia.org) [production]
10:39 <elukey> puppet disabled on rdb1004 (plus Redis instances) as precautionary step for master reimage - rdb1003 [production]
10:35 <jynus> setting up master-master replication on tools (labsdb100[45]) [production]
09:32 <moritzm> rebooting iron for kernel update [production]
09:26 <elukey> puppet disabled on rdb1003 for debian re-image. Redis servers going to be stopped too as pre-step for backup. [production]
08:32 <oblivian@tin> Synchronized wmf-config/jobqueue-eqiad.php: re-routing writes from rdb1003 to rdb1005 for reimaging (duration: 00m 36s) [production]
08:29 <jynus@tin> Synchronized wmf-config/db-codfw.php: Repool db2035 (duration: 00m 39s) [production]
07:49 <_joe_> uploading hhvm 3.12, extensions to reprepro [production]
06:26 <mutante> bast2001 - still installing in snail mode - please feel free to check if it's done, and if so re-add to puppet so users get created.thx [production]
03:16 <l10nupdate@tin> ResourceLoader cache refresh completed at Wed Mar 9 03:16:21 UTC 2016 (duration 8m 2s) [production]
03:08 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.16) (duration: 17m 51s) [production]
02:32 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.15) (duration: 13m 28s) [production]
02:13 <ori@tin> Synchronized php-1.27.0-wmf.15/includes/diff/DairikiDiff.php: I4d4b8f81c: Dont quote assert expressions in DairikiDiff (duration: 00m 27s) [production]
02:12 <ori@tin> Synchronized php-1.27.0-wmf.16/includes/diff/DairikiDiff.php: I4d4b8f81c: Dont quote assert expressions in DairikiDiff (duration: 00m 31s) [production]
01:40 <csteipp> redeploy patch for T129120 [production]
01:39 <csteipp@tin> Synchronized php-1.27.0-wmf.16/includes/api/ApiQueryInfo.php: (no message) (duration: 00m 33s) [production]
00:33 <maxsem@tin> Synchronized php-1.27.0-wmf.15/extensions/Echo/: SWAT (duration: 00m 32s) [production]
00:32 <maxsem@tin> Synchronized php-1.27.0-wmf.15/extensions/MobileFrontend/: SWAT (duration: 00m 32s) [production]
00:30 <maxsem@tin> Synchronized php-1.27.0-wmf.16/extensions/MobileFrontend/: SWAT (duration: 00m 29s) [production]
00:29 <maxsem@tin> Synchronized php-1.27.0-wmf.16/extensions/Echo/: SWAT (duration: 00m 38s) [production]
00:25 <maxsem@tin> Synchronized wmf-config/: https://gerrit.wikimedia.org/r/275378 - comment only (duration: 00m 27s) [production]
00:16 <maxsem@tin> Synchronized wmf-config/filebackend-production.php: https://gerrit.wikimedia.org/r/272922 (duration: 00m 27s) [production]
00:13 <maxsem@tin> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/275858/ (duration: 00m 28s) [production]
00:09 <maxsem@tin> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/275275 (duration: 00m 29s) [production]
2016-03-08 §
23:20 <mutante> bast2001 - install issues - extending downtime, bbiaw [production]
23:20 <twentyafterfour@tin> rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.27.0-wmf.16 [production]
23:11 <mutante> neon: re-enable puppet, start icinga-wm [production]
23:03 <twentyafterfour@tin> Synchronized w/static/: (no message) (duration: 00m 32s) [production]
22:53 <mutante> bast2001 - powercycle, reinstall [production]
22:19 <twentyafterfour@tin> Finished scap: testwiki to php-1.27.0-wmf.16 and rebuild l10n cache (duration: 26m 06s) [production]
22:04 <ori> Canary application servers (mw1017-mw1025) and canary API application servers (mw1114-mw1119) upgraded to HHVM 3.12.1 [production]
22:04 <jynus> recreating slave watchdog events on all servers [production]
21:53 <twentyafterfour@tin> Started scap: testwiki to php-1.27.0-wmf.16 and rebuild l10n cache [production]
21:02 <jynus@tin> Synchronized wmf-config/db-codfw.php: Repool db2040 (duration: 00m 34s) [production]
21:01 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Repool db1066 (duration: 00m 33s) [production]
20:49 <ori> Updated mw1200 to HHVM 3.12.1; repooling [production]
20:45 <ori> Depooled mw1200 for HHVM update [production]
20:40 <jynus> killing long running queries on db1066 [production]
20:39 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Depool db1066 (duration: 01m 49s) [production]
18:55 <mutante> temp. stopped icinga-wm [production]
18:54 <moritzm> uploaded linux 4.4.2-3+wmf1/jessie-wikimedia (based on Linux 4.4.4) to carbon [production]
17:46 <bblack> switching text cache routing: ulsfo->codfw [production]
17:36 <papaul> sinistra signing puppet certs, salt-key, initial run [production]