|
2023-06-21
ยง
|
| 22:35 |
<jclark@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt dbproxy1023 - jclark@cumin1001" |
[production] |
| 22:33 |
<jclark@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 22:33 |
<jclark@cumin1001> |
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dbproxy1023 |
[production] |
| 22:33 |
<jclark@cumin1001> |
START - Cookbook sre.network.configure-switch-interfaces for host dbproxy1023 |
[production] |
| 22:33 |
<jclark@cumin1001> |
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dbproxy1023 |
[production] |
| 22:33 |
<jclark@cumin1001> |
START - Cookbook sre.network.configure-switch-interfaces for host dbproxy1023 |
[production] |
| 22:26 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts gerrit1001.wikimedia.org |
[production] |
| 22:26 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 22:26 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: gerrit1001.wikimedia.org decommissioned, removing all IPs except the asset tag one - dzahn@cumin1001" |
[production] |
| 22:25 |
<dzahn@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: gerrit1001.wikimedia.org decommissioned, removing all IPs except the asset tag one - dzahn@cumin1001" |
[production] |
| 22:22 |
<dzahn@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 22:19 |
<eevans@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sessionstore2001.codfw.wmnet with OS bullseye |
[production] |
| 22:16 |
<mutante> |
destroying previous production gerrit server gerrit1001 - T336427 |
[production] |
| 22:15 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts gerrit1001.wikimedia.org |
[production] |
| 22:10 |
<mutante> |
rsyncing data from cobalt.wikimedia.org (:p) from gerrit1001 to gerrit1003, /srv/gerrit/cobalt/ |
[production] |
| 21:30 |
<eevans@cumin2002> |
START - Cookbook sre.hosts.reimage for host sessionstore2001.codfw.wmnet with OS bullseye |
[production] |
| 21:28 |
<eevans@cumin2002> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['sessionstore2001.codfw.wmnet'] |
[production] |
| 21:24 |
<eevans@cumin2002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sessionstore2001.codfw.wmnet'] |
[production] |
| 21:23 |
<eevans@cumin2002> |
END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['sessionstore2001.codfw.wmnet'] |
[production] |
| 21:23 |
<eevans@cumin2002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sessionstore2001.codfw.wmnet'] |
[production] |
| 21:23 |
<bking@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 20:52 |
<kostajh> |
UTC late deploys done |
[production] |
| 20:52 |
<kharlan@deploy1002> |
Finished scap: Backport for [[gerrit:931712|Section images: Select placeholder when inserting it (T335209)]] (duration: 10m 21s) |
[production] |
| 20:45 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts people2002.codfw.wmnet |
[production] |
| 20:45 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 20:45 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: people2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1001" |
[production] |
| 20:43 |
<kharlan@deploy1002> |
kharlan: Backport for [[gerrit:931712|Section images: Select placeholder when inserting it (T335209)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet |
[production] |
| 20:42 |
<kharlan@deploy1002> |
Started scap: Backport for [[gerrit:931712|Section images: Select placeholder when inserting it (T335209)]] |
[production] |
| 20:41 |
<dzahn@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: people2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1001" |
[production] |
| 20:36 |
<dzahn@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 20:30 |
<mutante> |
gerrit1001 (formerly gerrit prod) - creating tarball of entire /home/ in /home/ and copying it over to gerrit1003 - simultaneousy adding /home on gerrit servers to bacula from now on - T336427 |
[production] |
| 20:15 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts people2002.codfw.wmnet |
[production] |
| 20:14 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts people1003.eqiad.wmnet |
[production] |
| 20:14 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 20:14 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: people1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1001" |
[production] |
| 20:13 |
<dzahn@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: people1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1001" |
[production] |
| 20:09 |
<dzahn@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 20:04 |
<mutante> |
deleting VMs people1003.eqiad.wmnet and people2002.codfw.wmnet T338827 |
[production] |
| 20:03 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts people1003.eqiad.wmnet |
[production] |
| 19:59 |
<mutante> |
people.wikimedia.org - disabling shell access to people1003/people2002 (bullseye), use people1004/people2002 (bookworm) or people.eqiad.wmnet / people.codfw.wmnet in your configs if you have something automated or git repos - T338827 |
[production] |
| 19:28 |
<otto@deploy1002> |
helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 19:28 |
<otto@deploy1002> |
helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 19:24 |
<otto@deploy1002> |
helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 19:24 |
<otto@deploy1002> |
helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 19:22 |
<ejegg> |
civicrm upgraded from 4a4b014a to 98b2b5de |
[production] |
| 19:03 |
<otto@deploy1002> |
helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 19:03 |
<otto@deploy1002> |
helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 19:01 |
<sukhe@cumin2002> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['sessionstore2001.codfw.wmnet'] |
[production] |
| 19:01 |
<sukhe@cumin2002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sessionstore2001.codfw.wmnet'] |
[production] |
| 19:00 |
<eevans@cumin1001> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['sessionstore2001.codfw.wmnet'] |
[production] |