|
2019-07-03
§
|
| 09:47 |
<moritzm> |
rebooting debmonitor nodes to pick up MDS-enabled qemu |
[production] |
| 09:46 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:46 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:27 |
<moritzm> |
rebooting failoid nodes to pick up MDS-enabled qemu |
[production] |
| 09:25 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:25 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:01 |
<moritzm> |
rolling reboot of kubernetes masters in eqiad to pick up MDS-enabled qemu |
[production] |
| 08:44 |
<moritzm> |
rolling reboot of kubernetes masters in codfw to pick up MDS-enabled qemu |
[production] |
| 08:44 |
<moritzm> |
rolling reboot of kubernetes masters in codfw |
[production] |
| 08:43 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:43 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:45 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:45 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:34 |
<godog> |
reenable puppet fleetwide |
[production] |
| 07:33 |
<marostegui> |
Upgrade db2078 (s8 codfw master) |
[production] |
| 07:25 |
<marostegui> |
Upgrade db2100 (snapshots on that hosts are finished) |
[production] |
| 07:24 |
<godog> |
temporarily disable puppet to test/apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/520012 |
[production] |
| 07:23 |
<moritzm> |
updated buster installer d-i image to RC3 |
[production] |
| 07:10 |
<marostegui> |
Drop secret and scratch_tokens from labswiki (wikitech) and labstestwiki - T226826 |
[production] |
| 07:06 |
<marostegui> |
Drop secret and scratch_tokens from fishbowl wiki list T226826 |
[production] |
| 07:05 |
<godog> |
add 150G to graphite hosts lv, was at 94% utilization |
[production] |
| 06:55 |
<godog> |
depool and roll-restart swift proxy - T209182 |
[production] |
| 06:42 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Clarify db1069 status (duration: 00m 28s) |
[production] |
| 06:01 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Switchover x1 master eqiad from db1069 to db1120 T226358 (duration: 00m 27s) |
[production] |
| 06:00 |
<marostegui> |
Starting x1 failover from db1069 to db1120 - T226358 |
[production] |
| 06:00 |
<elukey> |
move the zookeeper puppet submodule into operations/puppet - T226466 |
[production] |
| 05:52 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 05:52 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 05:21 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 05:21 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 05:03 |
<vgutierrez> |
restarting pybal on lvs4006 |
[production] |
| 05:02 |
<marostegui> |
Start pre-failover steps for x1 - T226358 |
[production] |
| 04:47 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 04:47 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 04:34 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 04:34 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 04:24 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 04:24 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 04:23 |
<vgutierrez> |
rebooting primary lvs servers for MDS security updates |
[production] |
| 00:14 |
<eileen> |
process-control config revision is 8e215d07f2 (renable jobs) |
[production] |
| 00:08 |
<eileen> |
civicrm revision is 8a4451f390, config revision is ec8c43ee86 Redis |
[production] |
| 00:05 |
<eileen> |
process-control config revision is ec8c43ee86 (Redis turned on) |
[production] |
|
2019-07-02
§
|
| 23:42 |
<eileen> |
civicrm revision is 8a4451f390, config revision is c02a038331 (mysql locks enabled) |
[production] |
| 23:36 |
<catrope@deploy1001> |
Synchronized php-1.34.0-wmf.11/extensions/Echo/: T226594 (duration: 00m 51s) |
[production] |
| 23:34 |
<catrope@deploy1001> |
Synchronized php-1.34.0-wmf.11/skins/MonoBook/: T226594 (duration: 00m 50s) |
[production] |
| 22:35 |
<eileen> |
civicrm revision changed from 96985fcc4b to 8a4451f390, config revision is af9e657134 |
[production] |
| 20:35 |
<mutante> |
contint1001 - created new partitions on /dev/sdc and /dev/sdd; created new RAID 1 over /dev/sdc1 and /dev/sdd1 |
[production] |
| 20:28 |
<smalyshev@deploy1001> |
Finished deploy [wdqs/wdqs@cc60181]: Weekly WDQS deploy (duration: 14m 43s) |
[production] |
| 20:20 |
<mutante> |
contint1001 - temp installing parted for labeling new disks sdc and sdd for raid for docker images (T207707) |
[production] |
| 20:13 |
<smalyshev@deploy1001> |
Started deploy [wdqs/wdqs@cc60181]: Weekly WDQS deploy |
[production] |