|
2019-07-03
§
|
| 09:58 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:58 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:54 |
<moritzm> |
rebooting netmon2001 for kernel security update |
[production] |
| 09:52 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:52 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:47 |
<moritzm> |
rebooting debmonitor nodes to pick up MDS-enabled qemu |
[production] |
| 09:46 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:46 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:27 |
<moritzm> |
rebooting failoid nodes to pick up MDS-enabled qemu |
[production] |
| 09:25 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:25 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:01 |
<moritzm> |
rolling reboot of kubernetes masters in eqiad to pick up MDS-enabled qemu |
[production] |
| 08:44 |
<moritzm> |
rolling reboot of kubernetes masters in codfw to pick up MDS-enabled qemu |
[production] |
| 08:44 |
<moritzm> |
rolling reboot of kubernetes masters in codfw |
[production] |
| 08:43 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:43 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:45 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:45 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:34 |
<godog> |
reenable puppet fleetwide |
[production] |
| 07:33 |
<marostegui> |
Upgrade db2078 (s8 codfw master) |
[production] |
| 07:25 |
<marostegui> |
Upgrade db2100 (snapshots on that hosts are finished) |
[production] |
| 07:24 |
<godog> |
temporarily disable puppet to test/apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/520012 |
[production] |
| 07:23 |
<moritzm> |
updated buster installer d-i image to RC3 |
[production] |
| 07:10 |
<marostegui> |
Drop secret and scratch_tokens from labswiki (wikitech) and labstestwiki - T226826 |
[production] |
| 07:06 |
<marostegui> |
Drop secret and scratch_tokens from fishbowl wiki list T226826 |
[production] |
| 07:05 |
<godog> |
add 150G to graphite hosts lv, was at 94% utilization |
[production] |
| 06:55 |
<godog> |
depool and roll-restart swift proxy - T209182 |
[production] |
| 06:42 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Clarify db1069 status (duration: 00m 28s) |
[production] |
| 06:01 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Switchover x1 master eqiad from db1069 to db1120 T226358 (duration: 00m 27s) |
[production] |
| 06:00 |
<marostegui> |
Starting x1 failover from db1069 to db1120 - T226358 |
[production] |
| 06:00 |
<elukey> |
move the zookeeper puppet submodule into operations/puppet - T226466 |
[production] |
| 05:52 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 05:52 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 05:21 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 05:21 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 05:03 |
<vgutierrez> |
restarting pybal on lvs4006 |
[production] |
| 05:02 |
<marostegui> |
Start pre-failover steps for x1 - T226358 |
[production] |
| 04:47 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 04:47 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 04:34 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 04:34 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 04:24 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 04:24 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 04:23 |
<vgutierrez> |
rebooting primary lvs servers for MDS security updates |
[production] |
| 00:14 |
<eileen> |
process-control config revision is 8e215d07f2 (renable jobs) |
[production] |
| 00:08 |
<eileen> |
civicrm revision is 8a4451f390, config revision is ec8c43ee86 Redis |
[production] |
| 00:05 |
<eileen> |
process-control config revision is ec8c43ee86 (Redis turned on) |
[production] |