4301-4350 of 10000 results (84ms)
2024-02-23 ยง
18:55 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['db2210'] [production]
18:55 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['db2209'] [production]
18:55 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['db2209'] [production]
18:54 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['db2208'] [production]
18:54 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['db2208'] [production]
18:53 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['db2207'] [production]
18:53 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['db2200'] [production]
18:52 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['db2207'] [production]
18:52 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['db2207'] [production]
18:52 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['db2207'] [production]
18:52 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['db2206'] [production]
18:51 <jhancock@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['db2196'] [production]
18:51 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['db2206'] [production]
18:51 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['db2205'] [production]
18:51 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['db2205'] [production]
18:50 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['db2204'] [production]
18:50 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['db2204'] [production]
18:50 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['db2202'] [production]
18:50 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['db2202'] [production]
18:50 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['db2201'] [production]
18:49 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['db2201'] [production]
18:49 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['db2200'] [production]
18:49 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['db2200'] [production]
18:48 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['db2199'] [production]
18:48 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['db2199'] [production]
18:48 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['db2198'] [production]
18:47 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['db2198'] [production]
18:47 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['db2197'] [production]
18:47 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['db2197'] [production]
18:46 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['db2196'] [production]
18:46 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['db2196'] [production]
18:45 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['db2196'] [production]
18:43 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host db2203.mgmt.codfw.wmnet with reboot policy FORCED [production]
18:42 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:41 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
18:39 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db2203.mgmt.codfw.wmnet with reboot policy FORCED [production]
18:37 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host db2203.mgmt.codfw.wmnet with reboot policy FORCED [production]
18:14 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db1242 (T357189)', diff saved to https://phabricator.wikimedia.org/P57839 and previous config saved to /var/cache/conftool/dbconfig/20240223-181437-arnaudb.json [production]
18:14 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1242.eqiad.wmnet with reason: Maintenance [production]
18:14 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1242.eqiad.wmnet with reason: Maintenance [production]
18:14 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1241 (T357189)', diff saved to https://phabricator.wikimedia.org/P57838 and previous config saved to /var/cache/conftool/dbconfig/20240223-181416-arnaudb.json [production]
17:59 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P57835 and previous config saved to /var/cache/conftool/dbconfig/20240223-175909-arnaudb.json [production]
17:55 <Daimona> T357007 Running mwscript CampaignEvents:GenerateInvitationList --wiki=metawiki --listfile=/home/daimona/list.txt [production]
17:44 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P57834 and previous config saved to /var/cache/conftool/dbconfig/20240223-174403-arnaudb.json [production]
17:28 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1241 (T357189)', diff saved to https://phabricator.wikimedia.org/P57833 and previous config saved to /var/cache/conftool/dbconfig/20240223-172856-arnaudb.json [production]
16:24 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db1241 (T357189)', diff saved to https://phabricator.wikimedia.org/P57832 and previous config saved to /var/cache/conftool/dbconfig/20240223-162426-arnaudb.json [production]
16:24 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1241.eqiad.wmnet with reason: Maintenance [production]
16:23 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1241.eqiad.wmnet with reason: Maintenance [production]
16:23 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1221 (T357189)', diff saved to https://phabricator.wikimedia.org/P57831 and previous config saved to /var/cache/conftool/dbconfig/20240223-162351-arnaudb.json [production]
16:09 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp1100.eqiad.wmnet,service=(cdn|ats-be) [production]