2701-2750 of 10000 results (65ms)
2022-10-26 ยง
19:00 <sukhe@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4046.ulsfo.wmnet with OS buster [production]
18:45 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp4046.ulsfo.wmnet with OS buster [production]
18:41 <sukhe@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4046.ulsfo.wmnet with OS buster [production]
18:31 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp4046.ulsfo.wmnet with OS buster [production]
18:09 <sukhe@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4046.ulsfo.wmnet with OS buster [production]
18:07 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1166 (T321312)', diff saved to https://phabricator.wikimedia.org/P36611 and previous config saved to /var/cache/conftool/dbconfig/20221026-180742-ladsgroup.json [production]
18:07 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance [production]
18:07 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance [production]
18:07 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1157 (T321312)', diff saved to https://phabricator.wikimedia.org/P36610 and previous config saved to /var/cache/conftool/dbconfig/20221026-180718-ladsgroup.json [production]
18:01 <Amir1> dbmaint on s8@eqiad (T321562) [production]
18:01 <Amir1> dbmaint on s5@eqiad (T321562) [production]
18:00 <Amir1> dbmaint on s3@eqiad (T321562) [production]
18:00 <Amir1> dbmaint on s1@eqiad (T321562) [production]
17:54 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp4046.ulsfo.wmnet with OS buster [production]
17:52 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P36609 and previous config saved to /var/cache/conftool/dbconfig/20221026-175212-ladsgroup.json [production]
17:21 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1157 (T321312)', diff saved to https://phabricator.wikimedia.org/P36607 and previous config saved to /var/cache/conftool/dbconfig/20221026-172159-ladsgroup.json [production]
17:15 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1157 (T321312)', diff saved to https://phabricator.wikimedia.org/P36606 and previous config saved to /var/cache/conftool/dbconfig/20221026-171534-ladsgroup.json [production]
17:15 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance [production]
17:15 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance [production]
17:15 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1112 (T321312)', diff saved to https://phabricator.wikimedia.org/P36605 and previous config saved to /var/cache/conftool/dbconfig/20221026-171508-ladsgroup.json [production]
17:02 <hashar@deploy1002> Finished deploy [releng/phatality@d8dfa72]: Update Phatality on codfw for OpenSearch Dashboard 2.2.0 # T304440 (duration: 00m 27s) [production]
17:01 <hashar@deploy1002> Started deploy [releng/phatality@d8dfa72]: Update Phatality on codfw for OpenSearch Dashboard 2.2.0 # T304440 [production]
17:00 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P36604 and previous config saved to /var/cache/conftool/dbconfig/20221026-170001-ladsgroup.json [production]
16:54 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp4040.ulsfo.wmnet with OS buster [production]
16:44 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P36603 and previous config saved to /var/cache/conftool/dbconfig/20221026-164455-ladsgroup.json [production]
16:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1112 (T321312)', diff saved to https://phabricator.wikimedia.org/P36602 and previous config saved to /var/cache/conftool/dbconfig/20221026-162948-ladsgroup.json [production]
16:26 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
16:25 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
16:25 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1200 (T318950)', diff saved to https://phabricator.wikimedia.org/P36601 and previous config saved to /var/cache/conftool/dbconfig/20221026-162549-ladsgroup.json [production]
16:23 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1112 (T321312)', diff saved to https://phabricator.wikimedia.org/P36600 and previous config saved to /var/cache/conftool/dbconfig/20221026-162316-ladsgroup.json [production]
16:23 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1112.eqiad.wmnet with reason: Maintenance [production]
16:22 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1112.eqiad.wmnet with reason: Maintenance [production]
16:11 <sukhe@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4040.ulsfo.wmnet with OS buster [production]
16:11 <ladsgroup@cumin1001> END (ERROR) - Cookbook sre.ganeti.reboot-vm (exit_code=97) for VM lists1001.wikimedia.org [production]
16:11 <ladsgroup@cumin1001> END (ERROR) - Cookbook sre.ganeti.reboot-vm (exit_code=97) for VM lists1001.wikimedia.org [production]
16:10 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P36599 and previous config saved to /var/cache/conftool/dbconfig/20221026-161042-ladsgroup.json [production]
16:06 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
16:05 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
16:05 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
16:04 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:849626|Revert "kswiki: Switch to wikitext mentor provider back" (T310905)]] (duration: 04m 16s) [production]
16:04 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
16:03 <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
16:03 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dumpsdata1007.eqiad.wmnet [production]
16:03 <ladsgroup@cumin1001> START - Cookbook sre.ganeti.reboot-vm for VM lists1001.wikimedia.org [production]
16:02 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp4040.ulsfo.wmnet with OS buster [production]
16:00 <sukhe@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp4040 [production]
16:00 <sukhe@cumin2002> START - Cookbook sre.network.configure-switch-interfaces for host cp4040 [production]
16:00 <urbanecm@deploy1002> urbanecm and urbanecm: Backport for [[gerrit:849626|Revert "kswiki: Switch to wikitext mentor provider back" (T310905)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet [production]
16:00 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:849626|Revert "kswiki: Switch to wikitext mentor provider back" (T310905)]] [production]
15:59 <volans@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp4041.mgmt.ulsfo.wmnet with reboot policy FORCED [production]