2801-2850 of 10000 results (121ms)
2022-11-22 ยง
21:56 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
21:56 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1196 (T321130)', diff saved to https://phabricator.wikimedia.org/P40690 and previous config saved to /var/cache/conftool/dbconfig/20221122-215610-marostegui.json [production]
21:52 <samtar@deploy1002> samtar and jdlrobson: Backport for [[gerrit:859076|Update TOC to use PinnableHeader (T317897)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet [production]
21:52 <samtar@deploy1002> Started scap: Backport for [[gerrit:859076|Update TOC to use PinnableHeader (T317897)]] [production]
21:51 <samtar@deploy1002> Finished scap: Backport for [[gerrit:859508|Fix icon button spacing in sticky header (T323176)]] (duration: 07m 25s) [production]
21:44 <samtar@deploy1002> samtar and bwang: Backport for [[gerrit:859508|Fix icon button spacing in sticky header (T323176)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
21:44 <samtar@deploy1002> Started scap: Backport for [[gerrit:859508|Fix icon button spacing in sticky header (T323176)]] [production]
21:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P40689 and previous config saved to /var/cache/conftool/dbconfig/20221122-214103-marostegui.json [production]
21:33 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dbprov1004'] [production]
21:32 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dbprov1004.mgmt.eqiad.wmnet with reboot policy FORCED [production]
21:25 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P40688 and previous config saved to /var/cache/conftool/dbconfig/20221122-212556-marostegui.json [production]
21:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1196 (T321130)', diff saved to https://phabricator.wikimedia.org/P40687 and previous config saved to /var/cache/conftool/dbconfig/20221122-211049-marostegui.json [production]
21:04 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['puppetdb1003'] [production]
21:03 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host dbprov1004.mgmt.eqiad.wmnet with reboot policy FORCED [production]
21:03 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov1004.mgmt.eqiad.wmnet with reboot policy FORCED [production]
21:02 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host dbprov1004.mgmt.eqiad.wmnet with reboot policy FORCED [production]
21:01 <samtar@deploy1002> backport aborted: (duration: 00m 33s) [production]
20:58 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['puppetdb1003'] [production]
20:57 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['puppetdb1003'] [production]
20:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1196 (T321130)', diff saved to https://phabricator.wikimedia.org/P40686 and previous config saved to /var/cache/conftool/dbconfig/20221122-205720-marostegui.json [production]
20:57 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1196.eqiad.wmnet with reason: Maintenance [production]
20:57 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db1196.eqiad.wmnet with reason: Maintenance [production]
20:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1186 (T321130)', diff saved to https://phabricator.wikimedia.org/P40685 and previous config saved to /var/cache/conftool/dbconfig/20221122-205659-marostegui.json [production]
20:48 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['puppetdb1003'] [production]
20:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P40684 and previous config saved to /var/cache/conftool/dbconfig/20221122-204153-marostegui.json [production]
20:36 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host puppetdb1003.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P40683 and previous config saved to /var/cache/conftool/dbconfig/20221122-202646-marostegui.json [production]
20:23 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host puppetdb1003.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:21 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
20:19 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
20:11 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1186 (T321130)', diff saved to https://phabricator.wikimedia.org/P40682 and previous config saved to /var/cache/conftool/dbconfig/20221122-201140-marostegui.json [production]
20:07 <sukhe> sudo ipmitool -I lanplus -H "cp2041.mgmt.codfw.wmnet" -U root -E chassis power cycle [production]
20:05 <sukhe@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp2041'] [production]
20:05 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp2041'] [production]
20:05 <sukhe@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp2041'] [production]
20:04 <brett@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2041.codfw.wmnet with OS bullseye [production]
20:04 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp2041'] [production]
20:04 <sukhe@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp2041'] [production]
20:04 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp2041'] [production]
20:04 <sukhe@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp2041.codfw.wmnet'] [production]
20:04 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp2041.codfw.wmnet'] [production]
20:03 <sukhe@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp2041.codfw.wmnet'] [production]
20:03 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp2041.codfw.wmnet'] [production]
20:03 <sukhe@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp2041'] [production]
20:03 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp2041'] [production]
19:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1186 (T321130)', diff saved to https://phabricator.wikimedia.org/P40681 and previous config saved to /var/cache/conftool/dbconfig/20221122-195929-marostegui.json [production]
19:59 <sukhe@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp2041'] [production]
19:59 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1186.eqiad.wmnet with reason: Maintenance [production]
19:59 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp2041'] [production]
19:59 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db1186.eqiad.wmnet with reason: Maintenance [production]