|
2023-06-21
ยง
|
| 22:16 |
<mutante> |
destroying previous production gerrit server gerrit1001 - T336427 |
[production] |
| 22:15 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts gerrit1001.wikimedia.org |
[production] |
| 22:10 |
<mutante> |
rsyncing data from cobalt.wikimedia.org (:p) from gerrit1001 to gerrit1003, /srv/gerrit/cobalt/ |
[production] |
| 21:30 |
<eevans@cumin2002> |
START - Cookbook sre.hosts.reimage for host sessionstore2001.codfw.wmnet with OS bullseye |
[production] |
| 21:28 |
<eevans@cumin2002> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['sessionstore2001.codfw.wmnet'] |
[production] |
| 21:24 |
<eevans@cumin2002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sessionstore2001.codfw.wmnet'] |
[production] |
| 21:23 |
<eevans@cumin2002> |
END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['sessionstore2001.codfw.wmnet'] |
[production] |
| 21:23 |
<eevans@cumin2002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sessionstore2001.codfw.wmnet'] |
[production] |
| 21:23 |
<bking@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 20:52 |
<kostajh> |
UTC late deploys done |
[production] |
| 20:52 |
<kharlan@deploy1002> |
Finished scap: Backport for [[gerrit:931712|Section images: Select placeholder when inserting it (T335209)]] (duration: 10m 21s) |
[production] |
| 20:45 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts people2002.codfw.wmnet |
[production] |
| 20:45 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 20:45 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: people2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1001" |
[production] |
| 20:43 |
<kharlan@deploy1002> |
kharlan: Backport for [[gerrit:931712|Section images: Select placeholder when inserting it (T335209)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet |
[production] |
| 20:42 |
<kharlan@deploy1002> |
Started scap: Backport for [[gerrit:931712|Section images: Select placeholder when inserting it (T335209)]] |
[production] |
| 20:41 |
<dzahn@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: people2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1001" |
[production] |
| 20:36 |
<dzahn@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 20:30 |
<mutante> |
gerrit1001 (formerly gerrit prod) - creating tarball of entire /home/ in /home/ and copying it over to gerrit1003 - simultaneousy adding /home on gerrit servers to bacula from now on - T336427 |
[production] |
| 20:15 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts people2002.codfw.wmnet |
[production] |
| 20:14 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts people1003.eqiad.wmnet |
[production] |
| 20:14 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 20:14 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: people1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1001" |
[production] |
| 20:13 |
<dzahn@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: people1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1001" |
[production] |
| 20:09 |
<dzahn@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 20:04 |
<mutante> |
deleting VMs people1003.eqiad.wmnet and people2002.codfw.wmnet T338827 |
[production] |
| 20:03 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts people1003.eqiad.wmnet |
[production] |
| 19:59 |
<mutante> |
people.wikimedia.org - disabling shell access to people1003/people2002 (bullseye), use people1004/people2002 (bookworm) or people.eqiad.wmnet / people.codfw.wmnet in your configs if you have something automated or git repos - T338827 |
[production] |
| 19:28 |
<otto@deploy1002> |
helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 19:28 |
<otto@deploy1002> |
helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 19:24 |
<otto@deploy1002> |
helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 19:24 |
<otto@deploy1002> |
helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 19:22 |
<ejegg> |
civicrm upgraded from 4a4b014a to 98b2b5de |
[production] |
| 19:03 |
<otto@deploy1002> |
helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 19:03 |
<otto@deploy1002> |
helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 19:01 |
<sukhe@cumin2002> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['sessionstore2001.codfw.wmnet'] |
[production] |
| 19:01 |
<sukhe@cumin2002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sessionstore2001.codfw.wmnet'] |
[production] |
| 19:00 |
<eevans@cumin1001> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['sessionstore2001.codfw.wmnet'] |
[production] |
| 19:00 |
<eevans@cumin1001> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sessionstore2001.codfw.wmnet'] |
[production] |
| 18:48 |
<jclark@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy1025.eqiad.wmnet with OS bullseye |
[production] |
| 18:48 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1001" |
[production] |
| 18:29 |
<brett@cumin2002> |
START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp1078.eqiad.wmnet,cp1080.eqiad.wmnet,cp1082.eqiad.wmnet,cp1084.eqiad.wmnet,cp1086.eqiad.wmnet,cp1088.eqiad.wmnet,cp1090.eqiad.wmnet} and A:cp |
[production] |
| 18:27 |
<brett@cumin2002> |
START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp1077.eqiad.wmnet,cp1079.eqiad.wmnet,cp1081.eqiad.wmnet,cp1083.eqiad.wmnet,cp1085.eqiad.wmnet,cp1087.eqiad.wmnet,cp1089.eqiad.wmnet} and A:cp |
[production] |
| 18:24 |
<sukhe> |
sudo ipmitool -I lanplus -H "sessionstore2001.mgmt.codfw.wmnet" -U root -E mc reset cold |
[production] |
| 18:14 |
<sukhe@cumin2002> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['sessionstore2001.codfw.wmnet'] |
[production] |
| 18:13 |
<sukhe@cumin2002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sessionstore2001.codfw.wmnet'] |
[production] |
| 18:12 |
<sukhe@cumin2002> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['sessionstore2001.codfw.wmnet'] |
[production] |
| 18:12 |
<mutante> |
miscweb1003/miscweb2003 - rm -rf /srv/org/wikimedia/racktables T327405 |
[production] |
| 18:12 |
<sukhe@cumin2002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sessionstore2001.codfw.wmnet'] |
[production] |
| 18:09 |
<mutante> |
miscweb1003/miscweb2003 - rm -rf /srv/org/wikimedia/annualreport T337041 |
[production] |