251-300 of 10000 results (27ms)
2016-03-10 §
10:39 <mobrovac> restbase deploy end of 26bd4aa28 [production]
10:31 <jynus> setting up the rest of the cross-datacenter master-master connections pending in wmf databases [production]
10:09 <godog> decommissioning restbase1002.eqiad.wmnet : T125842 [production]
09:50 <volans@tin> Synchronized wmf-config/db-codfw.php: Rebalance external storage servers in codfw T127330 (duration: 00m 34s) [production]
09:50 <mobrovac> restbase deploy start of 26bd4aa28 on restbase1001 [production]
08:55 <ema> apache2 and hhvm restarted on mw1107, mw1122 and mw1119 [production]
03:08 <l10nupdate@tin> ResourceLoader cache refresh completed at Thu Mar 10 03:08:54 UTC 2016 (duration 9m 38s) [production]
02:59 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.16) (duration: 12m 33s) [production]
02:33 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.15) (duration: 14m 08s) [production]
00:45 <catrope@tin> Synchronized php-1.27.0-wmf.15/extensions/ZeroBanner: SWAT (duration: 00m 28s) [production]
00:44 <catrope@tin> Synchronized php-1.27.0-wmf.15/extensions/MobileFrontend: SWAT (duration: 00m 32s) [production]
00:39 <catrope@tin> Synchronized php-1.27.0-wmf.16/extensions/ZeroBanner: SWAT (duration: 00m 28s) [production]
00:39 <catrope@tin> Synchronized php-1.27.0-wmf.16/extensions/MobileFrontend: SWAT (duration: 00m 36s) [production]
00:30 <ori> Upgrade of HHVM package to 3.12.1+dfsg-1 complete on all eqiad hosts save terbium [production]
00:23 <catrope@tin> Synchronized wmf-config/InitialiseSettings.php: Set NS_PROJECT on azwiktionary (duration: 00m 29s) [production]
00:13 <catrope@tin> Synchronized wmf-config/CommonSettings.php: Remove svnadmins group (duration: 00m 41s) [production]
2016-03-09 §
23:54 <bblack> ending caches codfw->direct testing [production]
23:15 <bblack> restarted puppetmaster on palladium [production]
23:03 <bblack> caches: starting test codfw->direct (codfw caches -> eqiad apps) [production]
22:42 <bearND> mobileapps deployed 26d4031 [production]
22:36 <bearND> starting mobileapps deploy, second try (without dead link removal patch) [production]
21:54 <bearND> mobileapps deployed 95a2d76 [production]
21:28 <ori> Depooled mw1107 [production]
21:26 <bearND> starting mobileapps deploy [production]
21:22 <ori> restarted HHVM on mw1107; lock-up [production]
21:06 <twentyafterfour@tin> rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.16 [production]
20:57 <ori> Upgrade HHVM on CODFW app servers to 3.12.1+dfsg-1 [production]
20:19 <urandom> decommissioning restbase1001.eqiad.wmnet : T125842 [production]
20:11 <krinkle@tin> Synchronized wmf-config/db-eqiad.php: Clean up - Ibb4bb0b32f5 (duration: 00m 29s) [production]
20:11 <krinkle@tin> Synchronized wmf-config/db-codfw.php: Clean up - Ibb4bb0b32f5 (duration: 00m 35s) [production]
19:01 <volans@tin> Synchronized wmf-config/db-codfw.php: Change codfw external storage topology T127330 (duration: 00m 27s) [production]
18:52 <anomie@tin> Synchronized php-1.27.0-wmf.16/includes/session/SessionManager.php: Add backtrace to log for MW_NO_SESSION warning mode [[gerrit:276232]] (duration: 00m 50s) [production]
18:44 <volans> Changing topology of local codfw masters for es2 and es3 before merging https://gerrit.wikimedia.org/r/#/c/276229/1 T127330 [production]
17:53 <csteipp> deployed patch for T110143 to wmf16 [production]
17:40 <csteipp> deployed patch for T122056 [production]
16:09 <thcipriani@tin> Synchronized wmf-config/throttle.php: SWAT: Wikipedia while at Women of the World Festival throttle rule [[gerrit:276179]] (duration: 00m 30s) [production]
16:05 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: Change LanguageOverlay bucket rates [[gerrit:276172]] (duration: 00m 52s) [production]
14:54 <volans@tin> Synchronized wmf-config/db-codfw.php: Rebalance external storage servers in codfw T127330 (duration: 00m 41s) [production]
14:14 <moritzm> installing Django security updates [production]
13:19 <moritzm> installing beanshell security updates [production]
12:46 <paravoid> set disable on cr2-knams:xe-1/2/0 (Init7), issues with at least one large network [production]
12:25 <elukey> added rdb1003 back to the Job Runners queue. All the jobchron processes on jobrunners/videoscalers need to be restarted. [production]
12:07 <elukey@tin> Synchronized wmf-config/jobqueue-eqiad.php: Add rdb1003 back to the Redis JobQueue pool after maintenance (duration: 00m 34s) [production]
12:00 <elukey> re-enabled puppet on rdb1004 [production]
11:47 <moritzm> uploaded firmware-nonfree 20151018 to jessie-wikimedia on carbon [production]
11:39 <godog> rolling restart cassandra in staging after merging https://gerrit.wikimedia.org/r/#/c/275917/ [production]
11:27 <jynus> setting up master-master cross-datacenter replication for s2 (db1018-db2017) [production]
11:20 <ema> hhvm restarted on mw1140 [production]
11:04 <jynus> deployed manually new code at dbtree (noc.wikimedia.org) [production]
10:39 <elukey> puppet disabled on rdb1004 (plus Redis instances) as precautionary step for master reimage - rdb1003 [production]