|
2019-07-03
ยง
|
| 11:29 |
<moritzm> |
ran puppet clean/deactivate and debdeploy removal for cp3037 (host is broken for a long time and triggering failing Cumin/debdeploy runs) T227077 |
[production] |
| 11:14 |
<Urbanecm> |
EU SWAT done |
[production] |
| 11:14 |
<Urbanecm> |
Ran mwscript namespaceDupes.php --wiki=pawikisource --fix for T226959 |
[production] |
| 11:12 |
<urbanecm@deploy1001> |
Synchronized wmf-config/throttle.php: SWAT: [[:gerrit:520408|Add new throttle rule for enwiki event]] (T227059) (duration: 00m 48s) |
[production] |
| 11:11 |
<urbanecm@deploy1001> |
Synchronized wmf-config/throttle-analyze.php: SWAT: [[:gerrit:518298|[throttle-analyze] Grant autoconfirmed permission to user when throttle rule is applied]] (T204583) (duration: 00m 49s) |
[production] |
| 11:11 |
<moritzm> |
rebooting people1001 (people.wikimedia.org) to pick up MDS-enabled qemu |
[production] |
| 11:06 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[:gerrit:520174|Configuring Namespaces at pawikisource]] (T226959) (duration: 00m 52s) |
[production] |
| 11:05 |
<moritzm> |
rebooting krypton nodes to pick up MDS-enabled qemu |
[production] |
| 11:05 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:05 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:04 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:04 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:36 |
<Amir1> |
start of ladsgroup@mwmaint1002:~$ foreachwikiindblist wiktionary extensions/Cognate/maintenance/populateCognatePages.php (T226358) |
[production] |
| 10:12 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:11 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:11 |
<moritzm> |
rolling reboot of eventschema service hosts to pick up MDS-enabled qemu |
[production] |
| 10:00 |
<marostegui> |
Drop secret and stratch_tokens columns from the private wiki list T226826 |
[production] |
| 09:58 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:58 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:54 |
<moritzm> |
rebooting netmon2001 for kernel security update |
[production] |
| 09:52 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:52 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:47 |
<moritzm> |
rebooting debmonitor nodes to pick up MDS-enabled qemu |
[production] |
| 09:46 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:46 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:27 |
<moritzm> |
rebooting failoid nodes to pick up MDS-enabled qemu |
[production] |
| 09:25 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:25 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:01 |
<moritzm> |
rolling reboot of kubernetes masters in eqiad to pick up MDS-enabled qemu |
[production] |
| 08:44 |
<moritzm> |
rolling reboot of kubernetes masters in codfw to pick up MDS-enabled qemu |
[production] |
| 08:44 |
<moritzm> |
rolling reboot of kubernetes masters in codfw |
[production] |
| 08:43 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:43 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:45 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:45 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:34 |
<godog> |
reenable puppet fleetwide |
[production] |
| 07:33 |
<marostegui> |
Upgrade db2078 (s8 codfw master) |
[production] |
| 07:25 |
<marostegui> |
Upgrade db2100 (snapshots on that hosts are finished) |
[production] |
| 07:24 |
<godog> |
temporarily disable puppet to test/apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/520012 |
[production] |
| 07:23 |
<moritzm> |
updated buster installer d-i image to RC3 |
[production] |
| 07:10 |
<marostegui> |
Drop secret and scratch_tokens from labswiki (wikitech) and labstestwiki - T226826 |
[production] |
| 07:06 |
<marostegui> |
Drop secret and scratch_tokens from fishbowl wiki list T226826 |
[production] |
| 07:05 |
<godog> |
add 150G to graphite hosts lv, was at 94% utilization |
[production] |
| 06:55 |
<godog> |
depool and roll-restart swift proxy - T209182 |
[production] |
| 06:42 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Clarify db1069 status (duration: 00m 28s) |
[production] |
| 06:01 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Switchover x1 master eqiad from db1069 to db1120 T226358 (duration: 00m 27s) |
[production] |
| 06:00 |
<marostegui> |
Starting x1 failover from db1069 to db1120 - T226358 |
[production] |
| 06:00 |
<elukey> |
move the zookeeper puppet submodule into operations/puppet - T226466 |
[production] |
| 05:52 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 05:52 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |