|
2018-04-19
ยง
|
| 14:30 |
<marostegui> |
Star atop on db1114 without "-R" - T192551 |
[production] |
| 14:29 |
<marostegui> |
Deploy schema change on db1082 (this will generate lag on s5 on labs hosts) - T191519 T188299 T190148 |
[production] |
| 14:28 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1082 for alter table (duration: 01m 13s) |
[production] |
| 14:19 |
<ejegg> |
re-enabled queue jobs |
[production] |
| 14:14 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1113:3315 after alter table (duration: 01m 16s) |
[production] |
| 14:12 |
<jynus> |
starting reimage of db2074 |
[production] |
| 13:56 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2075, depool db2074 (duration: 01m 16s) |
[production] |
| 13:39 |
<marostegui> |
Stop atop on db1114 - T191996 |
[production] |
| 13:33 |
<marostegui> |
Start atop on db1114 - T191996 |
[production] |
| 13:30 |
<Trey314159> |
reindexing serbian wikis on elastic@eqiad (T189265) |
[production] |
| 13:30 |
<moritzm> |
upgrading mw1334-mw1337 (job runners) to MEMC_VAL_COMPRESSION_ZLIB enabled HHVM build |
[production] |
| 13:14 |
<reedy@tin> |
Synchronized wmf-config/InitialiseSettings.php: T192427 T189277 (duration: 01m 17s) |
[production] |
| 12:58 |
<jynus> |
starting reimage of db2075 |
[production] |
| 12:48 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Depool db2075 (duration: 01m 16s) |
[production] |
| 11:55 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2076 (duration: 01m 16s) |
[production] |
| 11:39 |
<moritzm> |
upgrading eqiad video scalers to MEMC_VAL_COMPRESSION_ZLIB enabled HHVM build |
[production] |
| 11:24 |
<marostegui> |
Run check_private_data on labsdb - T183566 |
[production] |
| 11:21 |
<marostegui> |
Sanitize lfnwiki - T183566 |
[production] |
| 11:20 |
<moritzm> |
upgrading app servers mw1238-mw1258 to MEMC_VAL_COMPRESSION_ZLIB enabled HHVM build |
[production] |
| 11:14 |
<akosiaris@tin> |
Synchronized wmf-config/ProductionServices.php: T181121 (duration: 01m 16s) |
[production] |
| 11:09 |
<akosiaris@tin> |
Synchronized wmf-config/ProductionServices.php: (no justification provided) (duration: 01m 17s) |
[production] |
| 11:05 |
<marostegui> |
Deploy schema change on db1113:3315 - T191519 T188299 T190148 |
[production] |
| 11:03 |
<jynus> |
starting reimage of db2076 |
[production] |
| 11:02 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1113:3315 for alter table (duration: 01m 16s) |
[production] |
| 11:01 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Depool db2076 (duration: 01m 18s) |
[production] |
| 10:34 |
<moritzm> |
upgrading API servers mw1221-mw1235 to MEMC_VAL_COMPRESSION_ZLIB enabled HHVM build |
[production] |
| 10:27 |
<vgutierrez> |
Repool (Re-enable BGP) lvs4005 - T191897 |
[production] |
| 09:59 |
<elukey> |
complete migration of zookeeper on conf100[123] |
[production] |
| 09:55 |
<akosiaris> |
reboot ganeti VMs on row_B in codfw for cache=none setting. T181121 |
[production] |
| 09:54 |
<vgutierrez> |
Updating puppet compiler facts |
[production] |
| 09:51 |
<moritzm> |
rolling restart of Cassandra on maps completed |
[production] |
| 09:33 |
<elukey> |
upgrade zookeper on conf100[123] from 3.4.5 to 3.4.9 - T182924 |
[production] |
| 09:31 |
<akosiaris> |
start a force puppet run in all of eqiad with a batch size of 30 |
[production] |
| 09:29 |
<akosiaris> |
stop ircecho for a while, puppetdb1001 reboot was eventful |
[production] |
| 09:17 |
<akosiaris> |
reboot puppetdb1001 for cache=none setting apply. T181121 |
[production] |
| 09:14 |
<moritzm> |
installing Java security updates on maps* plus rolling restart of Cassandra to pick up new JRE |
[production] |
| 09:06 |
<vgutierrez> |
Depool and reimage lvs4005 as stretch - T191897 |
[production] |
| 09:03 |
<moritzm> |
upgrading API server canaries to MEMC_VAL_COMPRESSION_ZLIB enabled HHVM build (T184854) |
[production] |
| 08:40 |
<vgutierrez> |
Repool (Re-enable BGP) lvs4006 - T191897 |
[production] |
| 08:14 |
<ema> |
reboot deploy1001 and arm keyholder T175288 |
[production] |
| 08:14 |
<moritzm> |
upgrading app server canaries to MEMC_VAL_COMPRESSION_ZLIB enabled HHVM build (T184854) |
[production] |
| 07:47 |
<akosiaris> |
set cache=none for ganeti VMs in codfw cluster configuration. VM reboots to follow T181121 |
[production] |
| 07:32 |
<vgutierrez> |
Depool and reimage lvs4006 - T191897 |
[production] |
| 07:24 |
<akosiaris> |
reboot ganeti VMs on row_A in eqiad for cache=none setting. T181121 |
[production] |
| 07:06 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1097:3315 after alter table (duration: 01m 17s) |
[production] |
| 05:36 |
<marostegui> |
Kill atop on db1114 - T191996 |
[production] |
| 05:33 |
<marostegui> |
Revert RX buffer changes on db1114 - T191996 |
[production] |
| 05:27 |
<marostegui> |
Deploy schema change on db1097:3315 - T191519 T188299 T190148 |
[production] |
| 05:26 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1097:3315 for alter table (duration: 01m 33s) |
[production] |
| 03:18 |
<urandom> |
decommissioning Cassandra, restbase1010-c -- T189822 |
[production] |