8551-8600 of 10000 results (144ms)
2022-11-29 ยง
20:30 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db2141.codfw.wmnet with reason: Maintenance [production]
20:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2130 (T321126)', diff saved to https://phabricator.wikimedia.org/P41793 and previous config saved to /var/cache/conftool/dbconfig/20221129-203040-marostegui.json [production]
20:17 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P41792 and previous config saved to /var/cache/conftool/dbconfig/20221129-201739-ladsgroup.json [production]
20:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P41791 and previous config saved to /var/cache/conftool/dbconfig/20221129-201533-marostegui.json [production]
20:02 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2182 (T323907)', diff saved to https://phabricator.wikimedia.org/P41790 and previous config saved to /var/cache/conftool/dbconfig/20221129-200233-ladsgroup.json [production]
20:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P41789 and previous config saved to /var/cache/conftool/dbconfig/20221129-200027-marostegui.json [production]
19:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2130 (T321126)', diff saved to https://phabricator.wikimedia.org/P41788 and previous config saved to /var/cache/conftool/dbconfig/20221129-194520-marostegui.json [production]
19:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2130 (T321126)', diff saved to https://phabricator.wikimedia.org/P41787 and previous config saved to /var/cache/conftool/dbconfig/20221129-194257-marostegui.json [production]
19:42 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2130.codfw.wmnet with reason: Maintenance [production]
19:42 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db2130.codfw.wmnet with reason: Maintenance [production]
19:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2116 (T321126)', diff saved to https://phabricator.wikimedia.org/P41786 and previous config saved to /var/cache/conftool/dbconfig/20221129-194235-marostegui.json [production]
19:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P41785 and previous config saved to /var/cache/conftool/dbconfig/20221129-192728-marostegui.json [production]
19:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2182 (T323907)', diff saved to https://phabricator.wikimedia.org/P41784 and previous config saved to /var/cache/conftool/dbconfig/20221129-192628-ladsgroup.json [production]
19:26 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2182.codfw.wmnet with reason: Maintenance [production]
19:26 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2182.codfw.wmnet with reason: Maintenance [production]
19:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T323907)', diff saved to https://phabricator.wikimedia.org/P41783 and previous config saved to /var/cache/conftool/dbconfig/20221129-192606-ladsgroup.json [production]
19:12 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P41782 and previous config saved to /var/cache/conftool/dbconfig/20221129-191220-marostegui.json [production]
19:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P41781 and previous config saved to /var/cache/conftool/dbconfig/20221129-191100-ladsgroup.json [production]
18:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2116 (T321126)', diff saved to https://phabricator.wikimedia.org/P41780 and previous config saved to /var/cache/conftool/dbconfig/20221129-185714-marostegui.json [production]
18:56 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P41779 and previous config saved to /var/cache/conftool/dbconfig/20221129-185553-ladsgroup.json [production]
18:56 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2116 (T321126)', diff saved to https://phabricator.wikimedia.org/P41778 and previous config saved to /var/cache/conftool/dbconfig/20221129-185450-marostegui.json [production]
18:56 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2116.codfw.wmnet with reason: Maintenance [production]
18:54 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db2116.codfw.wmnet with reason: Maintenance [production]
18:54 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2103 (T321126)', diff saved to https://phabricator.wikimedia.org/P41777 and previous config saved to /var/cache/conftool/dbconfig/20221129-185429-marostegui.json [production]
18:43 <sukhe> sukhe@cumin2002:~$ sudo ipmitool -I lanplus -H "cp5021.mgmt.eqsin.wmnet" -U root -E chassis power cycle [production]
18:42 <sukhe@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5021.eqsin.wmnet with OS buster [production]
18:40 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T323907)', diff saved to https://phabricator.wikimedia.org/P41776 and previous config saved to /var/cache/conftool/dbconfig/20221129-184047-ladsgroup.json [production]
18:39 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2103', diff saved to https://phabricator.wikimedia.org/P41775 and previous config saved to /var/cache/conftool/dbconfig/20221129-183922-marostegui.json [production]
18:36 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
18:36 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
18:36 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
18:32 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
18:28 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp5021.eqsin.wmnet with OS buster [production]
18:24 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2103', diff saved to https://phabricator.wikimedia.org/P41774 and previous config saved to /var/cache/conftool/dbconfig/20221129-182416-marostegui.json [production]
18:09 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2103 (T321126)', diff saved to https://phabricator.wikimedia.org/P41773 and previous config saved to /var/cache/conftool/dbconfig/20221129-180909-marostegui.json [production]
18:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2103 (T321126)', diff saved to https://phabricator.wikimedia.org/P41772 and previous config saved to /var/cache/conftool/dbconfig/20221129-180646-marostegui.json [production]
18:06 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2103.codfw.wmnet with reason: Maintenance [production]
18:06 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db2103.codfw.wmnet with reason: Maintenance [production]
18:05 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2102.codfw.wmnet with reason: Maintenance [production]
18:05 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db2102.codfw.wmnet with reason: Maintenance [production]
18:05 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2097.codfw.wmnet with reason: Maintenance [production]
18:05 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db2097.codfw.wmnet with reason: Maintenance [production]
18:05 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
18:04 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
18:04 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1196 (T321126)', diff saved to https://phabricator.wikimedia.org/P41771 and previous config saved to /var/cache/conftool/dbconfig/20221129-180451-marostegui.json [production]
18:04 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2169:3317 (T323907)', diff saved to https://phabricator.wikimedia.org/P41770 and previous config saved to /var/cache/conftool/dbconfig/20221129-180408-ladsgroup.json [production]
18:04 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2169.codfw.wmnet with reason: Maintenance [production]
18:03 <robh@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['dns5004'] [production]
18:03 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2169.codfw.wmnet with reason: Maintenance [production]
18:03 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2168:3317 (T323907)', diff saved to https://phabricator.wikimedia.org/P41769 and previous config saved to /var/cache/conftool/dbconfig/20221129-180347-ladsgroup.json [production]