5601-5650 of 10000 results (99ms)
2024-03-07 §
14:33 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db1167.eqiad.wmnet with reason: Maintenance [production]
14:32 <eevans@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on restbase1040.eqiad.wmnet with reason: Bootstrapping — T354560 [production]
14:32 <eevans@cumin1002> START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on restbase1040.eqiad.wmnet with reason: Bootstrapping — T354560 [production]
14:31 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2121.codfw.wmnet with reason: Maintenance [production]
14:31 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2121.codfw.wmnet with reason: Maintenance [production]
14:31 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1181.eqiad.wmnet with reason: Maintenance [production]
14:30 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db1181.eqiad.wmnet with reason: Maintenance [production]
14:29 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2107.codfw.wmnet with reason: Maintenance [production]
14:28 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2107.codfw.wmnet with reason: Maintenance [production]
14:28 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1162.eqiad.wmnet with reason: Maintenance [production]
14:28 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db1162.eqiad.wmnet with reason: Maintenance [production]
14:26 <btullis@deploy2002> helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply [production]
14:25 <btullis@deploy2002> helmfile [eqiad] START helmfile.d/services/editor-analytics: apply [production]
14:25 <btullis@deploy2002> helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply [production]
14:25 <btullis@deploy2002> helmfile [codfw] START helmfile.d/services/editor-analytics: apply [production]
14:25 <btullis@deploy2002> helmfile [staging] DONE helmfile.d/services/editor-analytics: apply [production]
14:25 <urbanecm@deploy2002> urbanecm and anzx: Continuing with sync [production]
14:24 <btullis@deploy2002> helmfile [staging] START helmfile.d/services/editor-analytics: apply [production]
14:24 <btullis@deploy2002> helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply [production]
14:24 <btullis@deploy2002> helmfile [eqiad] START helmfile.d/services/edit-analytics: apply [production]
14:24 <btullis@deploy2002> helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply [production]
14:23 <btullis@deploy2002> helmfile [codfw] START helmfile.d/services/edit-analytics: apply [production]
14:23 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host dbprov2006.mgmt.codfw.wmnet with reboot policy FORCED [production]
14:23 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host dbprov2005.mgmt.codfw.wmnet with reboot policy FORCED [production]
14:22 <urbanecm@deploy2002> urbanecm and anzx: Backport for [[gerrit:1008988|itwikivoyage: update wgNamespacesToBeSearchedDefault (T358456)]], [[gerrit:1008989|kowikisource: add NamespaceAliases for User and Usertalk namespaces (T358508)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
14:22 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:20 <urbanecm@deploy2002> Started scap: Backport for [[gerrit:1008988|itwikivoyage: update wgNamespacesToBeSearchedDefault (T358456)]], [[gerrit:1008989|kowikisource: add NamespaceAliases for User and Usertalk namespaces (T358508)]] [production]
14:20 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
14:20 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2006.mgmt.codfw.wmnet with reboot policy FORCED [production]
14:16 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2005.mgmt.codfw.wmnet with reboot policy FORCED [production]
14:15 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host dbprov2006.mgmt.codfw.wmnet with reboot policy FORCED [production]
14:14 <bking@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "sync hiera as instructed by failed reimage cookbook - bking@cumin2002 - T358727" [production]
14:13 <bking@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "sync hiera as instructed by failed reimage cookbook - bking@cumin2002 - T358727" [production]
14:11 <jnuche@deploy2002> Finished scap: Backport for [[gerrit:1009500|REST: allow lower-case method names]] (duration: 11m 40s) [production]
14:11 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1025.eqiad.wmnet with OS bullseye [production]
14:11 <bking@cumin2002> END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - bking@cumin2002" [production]
14:10 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2127.codfw.wmnet with reason: Maintenance [production]
14:09 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2127.codfw.wmnet with reason: Maintenance [production]
14:06 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1157.eqiad.wmnet with reason: Maintenance [production]
14:06 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db1157.eqiad.wmnet with reason: Maintenance [production]
14:01 <jnuche@deploy2002> jnuche: Continuing with sync [production]
14:01 <btullis@deploy2002> helmfile [staging] DONE helmfile.d/services/edit-analytics: apply [production]
14:01 <jnuche@deploy2002> jnuche: Backport for [[gerrit:1009500|REST: allow lower-case method names]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
14:00 <btullis@deploy2002> helmfile [staging] START helmfile.d/services/edit-analytics: apply [production]
13:59 <jnuche@deploy2002> Started scap: Backport for [[gerrit:1009500|REST: allow lower-case method names]] [production]
13:42 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1220 (re)pooling @ 100%: Reimaging + upgrade done', diff saved to https://phabricator.wikimedia.org/P58643 and previous config saved to /var/cache/conftool/dbconfig/20240307-134226-arnaudb.json [production]
13:36 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2179.codfw.wmnet with reason: Maintenance [production]
13:36 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2179.codfw.wmnet with reason: Maintenance [production]
13:36 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1238.eqiad.wmnet with reason: Maintenance [production]
13:36 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db1238.eqiad.wmnet with reason: Maintenance [production]