101-150 of 10000 results (75ms)
2023-11-30 ยง
19:49 <jclark@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['elastic1104'] [production]
19:48 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P54037 and previous config saved to /var/cache/conftool/dbconfig/20231130-194835-arnaudb.json [production]
19:41 <vriley@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ganeti1038'] [production]
19:37 <vriley@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ganeti1037'] [production]
19:37 <vriley@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ganeti1036'] [production]
19:34 <vriley@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ganeti1035'] [production]
19:33 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase2028.codfw.wmnet [production]
19:33 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P54036 and previous config saved to /var/cache/conftool/dbconfig/20231130-193329-arnaudb.json [production]
19:33 <vriley@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti1038'] [production]
19:31 <vriley@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti1037'] [production]
19:30 <vriley@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti1036'] [production]
19:30 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudrabbit1002.wikimedia.org with OS bookworm [production]
19:29 <vriley@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti1035'] [production]
19:29 <jclark@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['elastic1107'] [production]
19:28 <vriley@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ganeti1035'] [production]
19:28 <vriley@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti1035'] [production]
19:27 <jclark@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['elastic1103'] [production]
19:25 <eevans@cumin1001> START - Cookbook sre.hosts.reboot-single for host restbase2028.codfw.wmnet [production]
19:24 <jclark@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['elastic1104'] [production]
19:24 <jclark@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['elastic1104'] [production]
19:24 <jclark@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['elastic1104'] [production]
19:24 <jclark@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host elastic1104.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:22 <vriley@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['elastic1107'] [production]
19:22 <vriley@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['elastic1106'] [production]
19:22 <jclark@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['elastic1107'] [production]
19:21 <vriley@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['elastic1106'] [production]
19:21 <vriley@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['elastic1105'] [production]
19:20 <jclark@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host elastic1107.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:20 <vriley@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['elastic1105'] [production]
19:20 <vriley@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['elastic1103'] [production]
19:20 <jclark@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['elastic1103'] [production]
19:19 <jclark@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host elastic1103.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:19 <jclark@cumin1001> START - Cookbook sre.hosts.provision for host elastic1107.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:19 <jclark@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['elastic1105'] [production]
19:19 <jclark@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['elastic1106'] [production]
19:18 <jclark@cumin1001> START - Cookbook sre.hosts.provision for host elastic1103.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:18 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2149 (T348183)', diff saved to https://phabricator.wikimedia.org/P54035 and previous config saved to /var/cache/conftool/dbconfig/20231130-191822-arnaudb.json [production]
19:17 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host elastic1103.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:17 <jclark@cumin1001> START - Cookbook sre.hosts.provision for host elastic1103.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:15 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host elastic1103.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:15 <jclark@cumin1001> START - Cookbook sre.hosts.provision for host elastic1103.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:15 <jclark@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['elastic1107'] [production]
19:14 <jclark@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['elastic1107'] [production]
19:14 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host elastic1107.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:14 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host elastic1103.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:13 <jclark@cumin1001> START - Cookbook sre.hosts.provision for host elastic1103.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:13 <jclark@cumin1001> START - Cookbook sre.hosts.provision for host elastic1107.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:13 <jclark@cumin1001> START - Cookbook sre.hosts.provision for host elastic1104.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:13 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudrabbit1002.wikimedia.org with reason: host reimage [production]
19:12 <jclark@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['elastic1107'] [production]