1601-1650 of 10000 results (50ms)
2018-04-20 §
14:00 <jynus> upgrade and restart db2086 [production]
13:59 <jynus@tin> Synchronized wmf-config/db-codfw.php: Depool db2086 (duration: 01m 13s) [production]
13:35 <anomie> (re-)creating `slots` table on all wikis, following up T190153 and T184446#4143097 [production]
13:25 <moritzm> upgrading mysql (as shipped in Debian) on bohrium [production]
13:00 <moritzm> installing zsh security updates on trusty servers [production]
12:25 <moritzm> upgrading apache on auth* servers [production]
12:18 <jynus> upgrading and restarting dbstore2002 [production]
12:15 <jynus@tin> Synchronized wmf-config/db-codfw.php: Repool db2070 (duration: 01m 17s) [production]
12:06 <moritzm> installing apache security updates on video scalers [production]
12:05 <moritzm> upgrading apache on einsteinium/icinga.wikimedia.org [production]
11:53 <moritzm> installing apache security updates on netmon1002/2001 [production]
11:27 <elukey> reimage analytics1068 to Debian Stretch - T192557 [production]
11:06 <moritzm> installing tiff security updates on trusty [production]
09:58 <godog> upload scap 3.8.0-2 - T192124 [production]
09:51 <moritzm> upgrading deployment servers to MEMC_VAL_COMPRESSION_ZLIB enabled HHVM build [production]
09:41 <jynus> starting reimage of db2070 [production]
09:41 <moritzm> upgrading mwdebug servers to MEMC_VAL_COMPRESSION_ZLIB enabled HHVM build [production]
09:33 <jynus@tin> Synchronized wmf-config/db-codfw.php: Repool db2071, depool db2070 (duration: 01m 16s) [production]
09:12 <elukey> restart of mw apis showing ~50% cpu utilization as precaution before the weekend - mw[1224,1225,1228,1230,1231,1233-1235,1276-1283,1286,1312,1313,1315,1316,1341,1343,1344,1347,1348]* [production]
09:06 <moritzm> upgrading video scalers in codfw to MEMC_VAL_COMPRESSION_ZLIB enabled HHVM build [production]
08:41 <moritzm> upgrading job runners in codfw to MEMC_VAL_COMPRESSION_ZLIB enabled HHVM build [production]
08:39 <marostegui> Going to sanitize gorwiki euwikisource romdwikimedia inhwiki on db1095 - T189112 T189466 T187774 T184375 [production]
08:39 <elukey> restart hhvm on mw[1226,1232].eqiad.wmnet - high load [production]
08:05 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Fully repool db1114 in API - T191996 (duration: 01m 16s) [production]
07:57 <jynus> starting reimage of db2071 [production]
07:52 <jynus@tin> Synchronized wmf-config/db-codfw.php: Depool db2071 (duration: 01m 16s) [production]
07:48 <moritzm> upgrading app servers in codfw to MEMC_VAL_COMPRESSION_ZLIB enabled HHVM build [production]
07:40 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Give more API traffic to db1114 (duration: 01m 17s) [production]
07:38 <ema> cp3041: restart varnish-be due to mbox lag [production]
07:37 <akosiaris> upgrade qemu on ganeti2006 to 1:2.8+dfsg-3~bpo8+1 and migrate mwdebug2001 to it T150532 [production]
07:32 <ema> cp3030: restart varnish-be due to mbox lag [production]
07:30 <_joe_> upgrading hhvm on all jobrunners in eqiad [production]
07:13 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Slowly repool db1114 in API (duration: 01m 15s) [production]
07:09 <ema> cp3032/cp3043: restart varnish-be due to mbox lag [production]
07:08 <moritzm> upgrading API servers in codfw to MEMC_VAL_COMPRESSION_ZLIB enabled HHVM build [production]
07:06 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1110 after alter table (duration: 01m 16s) [production]
06:54 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Restore main traffic original weight for db1114 (duration: 01m 15s) [production]
06:26 <ema> kafka::analytics remove strongswan leftovers T185136 [production]
06:25 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Slowly repool db1114 (duration: 01m 15s) [production]
06:07 <marostegui> Stop mysql db1114 for a reboot [production]
06:07 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1114 (duration: 01m 16s) [production]
05:55 <_joe_> depooling mw1227 from live traffic for investigation [production]
05:31 <marostegui> Start atop on db1114 with "-R" option enabled - T192551 [production]
05:31 <marostegui> Deploy schema change on db1110 - T191519 T188299 T190148 [production]
05:30 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1110 for alter table (duration: 01m 17s) [production]
05:21 <ariel@tin> Finished deploy [dumps/dumps@c2d3bb4]: keep completed stubs/abstracts/logs files around for retries (duration: 00m 04s) [production]
05:20 <ariel@tin> Started deploy [dumps/dumps@c2d3bb4]: keep completed stubs/abstracts/logs files around for retries [production]
01:50 <krinkle@tin> Synchronized wmf-config/CommonSettings.php: If8fdce707d (duration: 01m 17s) [production]
2018-04-19 §
23:16 <ebernhardson@tin> Synchronized php-1.31.0-wmf.29/extensions/WikimediaEvents/modules/all/ext.wikimediaEvents.searchSatisfaction.js: SWAT: T187148: Turn off cirrus ab test (duration: 01m 18s) [production]
23:13 <ebernhardson@tin> Synchronized php-1.31.0-wmf.30/extensions/WikimediaEvents/modules/all/ext.wikimediaEvents.searchSatisfaction.js: SWAT: T187148: Turn off cirrus ab test (duration: 01m 17s) [production]