801-850 of 10000 results (106ms)
2025-05-19 §
05:48 <marostegui@cumin1002> dbctl commit (dc=all): 'db1210 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P76283 and previous config saved to /var/cache/conftool/dbconfig/20250519-054844-root.json [production]
05:42 <moritzm> uploaded openjdk-8 8u452-ga-1~deb12u1 to component/jdk8 for bookworm-wikimedia [production]
05:33 <marostegui@cumin1002> dbctl commit (dc=all): 'db1210 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P76282 and previous config saved to /var/cache/conftool/dbconfig/20250519-053338-root.json [production]
05:18 <marostegui@cumin1002> dbctl commit (dc=all): 'db1210 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P76281 and previous config saved to /var/cache/conftool/dbconfig/20250519-051832-root.json [production]
05:12 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1210.eqiad.wmnet with reason: Maintenance [production]
05:12 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1210 T394508', diff saved to https://phabricator.wikimedia.org/P76280 and previous config saved to /var/cache/conftool/dbconfig/20250519-051224-marostegui.json [production]
04:59 <marostegui> Deploy schema change in x1 eqiad (with replication) dbmaint T394509 [production]
2025-05-18 §
12:50 <sbassett> Ran scap remove-patch for T392976 [production]
2025-05-17 §
17:33 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host pc2018.codfw.wmnet with OS bookworm [production]
17:33 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host cloudcontrol2010-dev.codfw.wmnet with OS bookworm [production]
17:33 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['pc2018'] [production]
17:33 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['sretest2004'] [production]
17:32 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sretest2004'] [production]
17:32 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['pc2018'] [production]
17:25 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:25 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pc2018.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:13 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:11 <jhancock@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcontrol2010-dev'] [production]
17:08 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['es2048'] [production]
17:08 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['es2047'] [production]
17:08 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['es2048'] [production]
17:08 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['es2047'] [production]
17:07 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host es2048.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:01 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host es2047.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
16:59 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcontrol2010-dev'] [production]
16:59 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host sretest2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
16:59 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcontrol2010-dev.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
16:58 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host pc2018.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
16:57 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
16:57 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be2009.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
16:53 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host es2048.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
16:51 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be2008.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
16:49 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host es2047.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
16:48 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host cloudcontrol2010-dev.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
16:47 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host sretest2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
16:33 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
16:33 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
16:32 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
16:22 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host thanos-be2009.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
15:58 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host apus-be2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
15:55 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host thanos-be2008.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
15:47 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host thanos-be2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
15:33 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host thanos-be2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
15:25 <cgoubert@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [production]
15:25 <cgoubert@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [production]
15:24 <cgoubert@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [production]
15:24 <cgoubert@deploy1003> helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [production]
15:20 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host apus-be2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
15:18 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host sretest2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
15:17 <jhancock@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host sretest2004 [production]