|
2018-04-20
§
|
| 13:25 |
<moritzm> |
upgrading mysql (as shipped in Debian) on bohrium |
[production] |
| 13:00 |
<moritzm> |
installing zsh security updates on trusty servers |
[production] |
| 12:25 |
<moritzm> |
upgrading apache on auth* servers |
[production] |
| 12:18 |
<jynus> |
upgrading and restarting dbstore2002 |
[production] |
| 12:15 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2070 (duration: 01m 17s) |
[production] |
| 12:06 |
<moritzm> |
installing apache security updates on video scalers |
[production] |
| 12:05 |
<moritzm> |
upgrading apache on einsteinium/icinga.wikimedia.org |
[production] |
| 11:53 |
<moritzm> |
installing apache security updates on netmon1002/2001 |
[production] |
| 11:27 |
<elukey> |
reimage analytics1068 to Debian Stretch - T192557 |
[production] |
| 11:06 |
<moritzm> |
installing tiff security updates on trusty |
[production] |
| 09:58 |
<godog> |
upload scap 3.8.0-2 - T192124 |
[production] |
| 09:51 |
<moritzm> |
upgrading deployment servers to MEMC_VAL_COMPRESSION_ZLIB enabled HHVM build |
[production] |
| 09:41 |
<jynus> |
starting reimage of db2070 |
[production] |
| 09:41 |
<moritzm> |
upgrading mwdebug servers to MEMC_VAL_COMPRESSION_ZLIB enabled HHVM build |
[production] |
| 09:33 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2071, depool db2070 (duration: 01m 16s) |
[production] |
| 09:12 |
<elukey> |
restart of mw apis showing ~50% cpu utilization as precaution before the weekend - mw[1224,1225,1228,1230,1231,1233-1235,1276-1283,1286,1312,1313,1315,1316,1341,1343,1344,1347,1348]* |
[production] |
| 09:06 |
<moritzm> |
upgrading video scalers in codfw to MEMC_VAL_COMPRESSION_ZLIB enabled HHVM build |
[production] |
| 08:41 |
<moritzm> |
upgrading job runners in codfw to MEMC_VAL_COMPRESSION_ZLIB enabled HHVM build |
[production] |
| 08:39 |
<marostegui> |
Going to sanitize gorwiki euwikisource romdwikimedia inhwiki on db1095 - T189112 T189466 T187774 T184375 |
[production] |
| 08:39 |
<elukey> |
restart hhvm on mw[1226,1232].eqiad.wmnet - high load |
[production] |
| 08:05 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Fully repool db1114 in API - T191996 (duration: 01m 16s) |
[production] |
| 07:57 |
<jynus> |
starting reimage of db2071 |
[production] |
| 07:52 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Depool db2071 (duration: 01m 16s) |
[production] |
| 07:48 |
<moritzm> |
upgrading app servers in codfw to MEMC_VAL_COMPRESSION_ZLIB enabled HHVM build |
[production] |
| 07:40 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Give more API traffic to db1114 (duration: 01m 17s) |
[production] |
| 07:38 |
<ema> |
cp3041: restart varnish-be due to mbox lag |
[production] |
| 07:37 |
<akosiaris> |
upgrade qemu on ganeti2006 to 1:2.8+dfsg-3~bpo8+1 and migrate mwdebug2001 to it T150532 |
[production] |
| 07:32 |
<ema> |
cp3030: restart varnish-be due to mbox lag |
[production] |
| 07:30 |
<_joe_> |
upgrading hhvm on all jobrunners in eqiad |
[production] |
| 07:13 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Slowly repool db1114 in API (duration: 01m 15s) |
[production] |
| 07:09 |
<ema> |
cp3032/cp3043: restart varnish-be due to mbox lag |
[production] |
| 07:08 |
<moritzm> |
upgrading API servers in codfw to MEMC_VAL_COMPRESSION_ZLIB enabled HHVM build |
[production] |
| 07:06 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1110 after alter table (duration: 01m 16s) |
[production] |
| 06:54 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Restore main traffic original weight for db1114 (duration: 01m 15s) |
[production] |
| 06:26 |
<ema> |
kafka::analytics remove strongswan leftovers T185136 |
[production] |
| 06:25 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Slowly repool db1114 (duration: 01m 15s) |
[production] |
| 06:07 |
<marostegui> |
Stop mysql db1114 for a reboot |
[production] |
| 06:07 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1114 (duration: 01m 16s) |
[production] |
| 05:55 |
<_joe_> |
depooling mw1227 from live traffic for investigation |
[production] |
| 05:31 |
<marostegui> |
Start atop on db1114 with "-R" option enabled - T192551 |
[production] |
| 05:31 |
<marostegui> |
Deploy schema change on db1110 - T191519 T188299 T190148 |
[production] |
| 05:30 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1110 for alter table (duration: 01m 17s) |
[production] |
| 05:21 |
<ariel@tin> |
Finished deploy [dumps/dumps@c2d3bb4]: keep completed stubs/abstracts/logs files around for retries (duration: 00m 04s) |
[production] |
| 05:20 |
<ariel@tin> |
Started deploy [dumps/dumps@c2d3bb4]: keep completed stubs/abstracts/logs files around for retries |
[production] |
| 01:50 |
<krinkle@tin> |
Synchronized wmf-config/CommonSettings.php: If8fdce707d (duration: 01m 17s) |
[production] |