|
2023-05-22
ยง
|
| 15:56 |
<bking@cumin1001> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2009.codfw.wmnet |
[production] |
| 15:32 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling update on A:netbox |
[production] |
| 15:26 |
<ayounsi@cumin1001> |
START - Cookbook sre.netbox.update-extras rolling update on A:netbox |
[production] |
| 15:25 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling update on A:netbox-canary |
[production] |
| 15:25 |
<ayounsi@cumin1001> |
START - Cookbook sre.netbox.update-extras rolling update on A:netbox-canary |
[production] |
| 15:12 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "New debmonitor VMs - jmm@cumin2002 - T241049" |
[production] |
| 15:10 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "New debmonitor VMs - jmm@cumin2002 - T241049" |
[production] |
| 14:32 |
<bking@deploy1002> |
helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 14:31 |
<bking@deploy1002> |
helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 14:10 |
<bking@deploy1002> |
helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 14:10 |
<bking@deploy1002> |
helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 12:57 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host debmonitor2003.codfw.wmnet with OS bookworm |
[production] |
| 12:44 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on debmonitor2003.codfw.wmnet with reason: host reimage |
[production] |
| 12:40 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on debmonitor2003.codfw.wmnet with reason: host reimage |
[production] |
| 12:20 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reimage for host debmonitor2003.codfw.wmnet with OS bookworm |
[production] |
| 12:19 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host debmonitor1003.eqiad.wmnet with OS bookworm |
[production] |
| 12:02 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on debmonitor1003.eqiad.wmnet with reason: host reimage |
[production] |
| 11:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2124', diff saved to https://phabricator.wikimedia.org/P48456 and previous config saved to /var/cache/conftool/dbconfig/20230522-115936-root.json |
[production] |
| 11:57 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on debmonitor1003.eqiad.wmnet with reason: host reimage |
[production] |
| 11:45 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reimage for host debmonitor1003.eqiad.wmnet with OS bookworm |
[production] |
| 10:17 |
<topranks> |
Un-draining transport circuit from eqsin to codfw, moving traffic back to default path T337220 |
[production] |
| 10:17 |
<topranks> |
Un-draining transport circuit from eqsin to codfw, moving traffic back to default path |
[production] |
| 10:06 |
<hashar@deploy1002> |
Finished scap: Backport for [[gerrit:921558|Revert "[WikibaseMediaInfo] Add 'main subject of' property"]] (duration: 37m 00s) |
[production] |
| 10:06 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host debmonitor2003.codfw.wmnet |
[production] |
| 10:06 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM debmonitor2003.codfw.wmnet - jmm@cumin2002" |
[production] |
| 10:05 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM debmonitor2003.codfw.wmnet - jmm@cumin2002" |
[production] |
| 10:04 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) debmonitor2003.codfw.wmnet on all recursors |
[production] |
| 10:04 |
<jmm@cumin2002> |
START - Cookbook sre.dns.wipe-cache debmonitor2003.codfw.wmnet on all recursors |
[production] |
| 10:04 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 10:04 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM debmonitor2003.codfw.wmnet - jmm@cumin2002" |
[production] |
| 10:03 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM debmonitor2003.codfw.wmnet - jmm@cumin2002" |
[production] |
| 10:02 |
<moritzm> |
installing updated usb.ids packages for Bullseye |
[production] |
| 10:01 |
<jmm@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 10:01 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.makevm for new host debmonitor2003.codfw.wmnet |
[production] |
| 09:51 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host debmonitor1003.eqiad.wmnet |
[production] |
| 09:51 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM debmonitor1003.eqiad.wmnet - jmm@cumin2002" |
[production] |
| 09:50 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM debmonitor1003.eqiad.wmnet - jmm@cumin2002" |
[production] |
| 09:49 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) debmonitor1003.eqiad.wmnet on all recursors |
[production] |
| 09:49 |
<jmm@cumin2002> |
START - Cookbook sre.dns.wipe-cache debmonitor1003.eqiad.wmnet on all recursors |
[production] |
| 09:49 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 09:49 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM debmonitor1003.eqiad.wmnet - jmm@cumin2002" |
[production] |
| 09:48 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM debmonitor1003.eqiad.wmnet - jmm@cumin2002" |
[production] |
| 09:43 |
<jmm@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 09:43 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.makevm for new host debmonitor1003.eqiad.wmnet |
[production] |
| 09:39 |
<hashar@deploy1002> |
hashar: Backport for [[gerrit:921558|Revert "[WikibaseMediaInfo] Add 'main subject of' property"]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 09:29 |
<hashar@deploy1002> |
Started scap: Backport for [[gerrit:921558|Revert "[WikibaseMediaInfo] Add 'main subject of' property"]] |
[production] |
| 08:46 |
<marostegui> |
Stop mysql on db2160 (haproxy irc alerts will be generated) |
[production] |
| 08:28 |
<elukey> |
drain Arelion link between cr1-codfw and cr3-eqsin to mitigate packet loss eqiad <-> eqsin |
[production] |
| 08:22 |
<moritzm> |
installing systemd security updates |
[production] |
| 08:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es2023 (re)pooling @ 100%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48455 and previous config saved to /var/cache/conftool/dbconfig/20230522-081724-root.json |
[production] |