|
2023-05-22
ยง
|
| 11:57 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on debmonitor1003.eqiad.wmnet with reason: host reimage |
[production] |
| 11:45 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reimage for host debmonitor1003.eqiad.wmnet with OS bookworm |
[production] |
| 10:17 |
<topranks> |
Un-draining transport circuit from eqsin to codfw, moving traffic back to default path T337220 |
[production] |
| 10:17 |
<topranks> |
Un-draining transport circuit from eqsin to codfw, moving traffic back to default path |
[production] |
| 10:06 |
<hashar@deploy1002> |
Finished scap: Backport for [[gerrit:921558|Revert "[WikibaseMediaInfo] Add 'main subject of' property"]] (duration: 37m 00s) |
[production] |
| 10:06 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host debmonitor2003.codfw.wmnet |
[production] |
| 10:06 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM debmonitor2003.codfw.wmnet - jmm@cumin2002" |
[production] |
| 10:05 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM debmonitor2003.codfw.wmnet - jmm@cumin2002" |
[production] |
| 10:04 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) debmonitor2003.codfw.wmnet on all recursors |
[production] |
| 10:04 |
<jmm@cumin2002> |
START - Cookbook sre.dns.wipe-cache debmonitor2003.codfw.wmnet on all recursors |
[production] |
| 10:04 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 10:04 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM debmonitor2003.codfw.wmnet - jmm@cumin2002" |
[production] |
| 10:03 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM debmonitor2003.codfw.wmnet - jmm@cumin2002" |
[production] |
| 10:02 |
<moritzm> |
installing updated usb.ids packages for Bullseye |
[production] |
| 10:01 |
<jmm@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 10:01 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.makevm for new host debmonitor2003.codfw.wmnet |
[production] |
| 09:51 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host debmonitor1003.eqiad.wmnet |
[production] |
| 09:51 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM debmonitor1003.eqiad.wmnet - jmm@cumin2002" |
[production] |
| 09:50 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM debmonitor1003.eqiad.wmnet - jmm@cumin2002" |
[production] |
| 09:49 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) debmonitor1003.eqiad.wmnet on all recursors |
[production] |
| 09:49 |
<jmm@cumin2002> |
START - Cookbook sre.dns.wipe-cache debmonitor1003.eqiad.wmnet on all recursors |
[production] |
| 09:49 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 09:49 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM debmonitor1003.eqiad.wmnet - jmm@cumin2002" |
[production] |
| 09:48 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM debmonitor1003.eqiad.wmnet - jmm@cumin2002" |
[production] |
| 09:43 |
<jmm@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 09:43 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.makevm for new host debmonitor1003.eqiad.wmnet |
[production] |
| 09:39 |
<hashar@deploy1002> |
hashar: Backport for [[gerrit:921558|Revert "[WikibaseMediaInfo] Add 'main subject of' property"]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 09:29 |
<hashar@deploy1002> |
Started scap: Backport for [[gerrit:921558|Revert "[WikibaseMediaInfo] Add 'main subject of' property"]] |
[production] |
| 08:46 |
<marostegui> |
Stop mysql on db2160 (haproxy irc alerts will be generated) |
[production] |
| 08:28 |
<elukey> |
drain Arelion link between cr1-codfw and cr3-eqsin to mitigate packet loss eqiad <-> eqsin |
[production] |
| 08:22 |
<moritzm> |
installing systemd security updates |
[production] |
| 08:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es2023 (re)pooling @ 100%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48455 and previous config saved to /var/cache/conftool/dbconfig/20230522-081724-root.json |
[production] |
| 08:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es2023 (re)pooling @ 75%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48454 and previous config saved to /var/cache/conftool/dbconfig/20230522-080219-root.json |
[production] |
| 07:59 |
<elukey> |
restart purged on cp5017 as test to clear out consumer group timeouts and rejoin events |
[production] |
| 07:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es2021 (re)pooling @ 100%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48453 and previous config saved to /var/cache/conftool/dbconfig/20230522-075613-root.json |
[production] |
| 07:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es2023 (re)pooling @ 50%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48452 and previous config saved to /var/cache/conftool/dbconfig/20230522-074715-root.json |
[production] |
| 07:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es2021 (re)pooling @ 75%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48451 and previous config saved to /var/cache/conftool/dbconfig/20230522-074109-root.json |
[production] |
| 07:37 |
<mvernon@cumin1001> |
END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:codfw and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) |
[production] |
| 07:32 |
<mvernon@cumin1001> |
START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:codfw and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) |
[production] |
| 07:32 |
<mvernon@cumin1001> |
END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:eqiad and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) |
[production] |
| 07:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es2023 (re)pooling @ 25%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48450 and previous config saved to /var/cache/conftool/dbconfig/20230522-073210-root.json |
[production] |
| 07:28 |
<mvernon@cumin1001> |
START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:eqiad and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) |
[production] |
| 07:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es2021 (re)pooling @ 50%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48449 and previous config saved to /var/cache/conftool/dbconfig/20230522-072604-root.json |
[production] |
| 07:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es2023 (re)pooling @ 10%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48448 and previous config saved to /var/cache/conftool/dbconfig/20230522-071705-root.json |
[production] |
| 07:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1031 (re)pooling @ 100%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48447 and previous config saved to /var/cache/conftool/dbconfig/20230522-071333-root.json |
[production] |
| 07:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1030 (re)pooling @ 100%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48446 and previous config saved to /var/cache/conftool/dbconfig/20230522-071326-root.json |
[production] |
| 07:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1029 (re)pooling @ 100%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48445 and previous config saved to /var/cache/conftool/dbconfig/20230522-071319-root.json |
[production] |
| 07:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es2021 (re)pooling @ 25%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48444 and previous config saved to /var/cache/conftool/dbconfig/20230522-071059-root.json |
[production] |
| 07:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es2023 (re)pooling @ 5%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48443 and previous config saved to /var/cache/conftool/dbconfig/20230522-070200-root.json |
[production] |
| 06:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1031 (re)pooling @ 75%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48442 and previous config saved to /var/cache/conftool/dbconfig/20230522-065828-root.json |
[production] |