7151-7200 of 10000 results (99ms)
2023-04-11 §
05:45 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Swakiyama out of all services on: 814 hosts [production]
05:45 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Swakiyama out of all services on: 814 hosts [production]
05:44 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Swakiyama out of all services on: 1241 hosts [production]
05:43 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Swakiyama out of all services on: 1241 hosts [production]
04:10 <eileen> civicrm upgraded from bc2f5ccc to b573aee4 [production]
03:54 <mwpresync@deploy2002> Pruned MediaWiki: 1.41.0-wmf.2 (duration: 02m 15s) [production]
03:52 <mwpresync@deploy2002> Finished scap: testwikis wikis to 1.41.0-wmf.4 refs T330210 (duration: 49m 57s) [production]
03:02 <mwpresync@deploy2002> Started scap: testwikis wikis to 1.41.0-wmf.4 refs T330210 [production]
00:37 <eileen> civicrm upgraded from 001e156a to bc2f5ccc [production]
00:13 <eileen> civicrm upgraded from 223f655a to 001e156a [production]
2023-04-10 §
23:07 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts miscweb1002.eqiad.wmnet [production]
23:07 <dzahn@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
23:07 <dzahn@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: miscweb1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1001" [production]
23:06 <dzahn@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: miscweb1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1001" [production]
23:00 <dzahn@cumin1001> START - Cookbook sre.dns.netbox [production]
22:55 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission for hosts miscweb1002.eqiad.wmnet [production]
22:53 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on miscweb1002.eqiad.wmnet with reason: decom [production]
22:53 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on miscweb1002.eqiad.wmnet with reason: decom [production]
21:53 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs3005.esams.wmnet with OS bullseye [production]
21:46 <urandom> restarting Cassandra, sessionstore1001-a, to restore native transport settings — T327954 [production]
21:36 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs3005.esams.wmnet with reason: host reimage [production]
21:33 <eevans@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host sessionstore1001.eqiad.wmnet [production]
21:32 <brett@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on lvs3005.esams.wmnet with reason: host reimage [production]
21:31 <urandom> restarting Cassandra, sessionstore1002-a — T327954 [production]
21:22 <eevans@cumin1001> START - Cookbook sre.hosts.reboot-single for host sessionstore1001.eqiad.wmnet [production]
21:21 <eevans@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host sessionstore1001.eqiad.wmnet [production]
21:14 <brett@cumin2002> START - Cookbook sre.hosts.reimage for host lvs3005.esams.wmnet with OS bullseye [production]
21:14 <brett@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs3005.esams.wmnet with OS bullseye [production]
21:13 <sbassett> Deployed updated security mitigation for T333140 [production]
21:10 <eevans@cumin1001> START - Cookbook sre.hosts.reboot-single for host sessionstore1001.eqiad.wmnet [production]
21:08 <eevans@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host sessionstore1001.eqiad.wmnet [production]
21:06 <urandom> restarting Cassandra, sessionstore1003-a — T327954 [production]
21:04 <urandom> restarting Cassandra, sessionstore1002-a — T327954 [production]
20:57 <eevans@cumin1001> START - Cookbook sre.hosts.reboot-single for host sessionstore1001.eqiad.wmnet [production]
20:40 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs3005.esams.wmnet with reason: host reimage [production]
20:36 <brett@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on lvs3005.esams.wmnet with reason: host reimage [production]
20:15 <brett@cumin2002> START - Cookbook sre.hosts.reimage for host lvs3005.esams.wmnet with OS bullseye [production]
20:09 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1073.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:07 <jclark@cumin1001> START - Cookbook sre.hosts.provision for host ms-be1073.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:07 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1072.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:05 <jclark@cumin1001> START - Cookbook sre.hosts.provision for host ms-be1072.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:53 <jclark@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cloudvirtlocal1003'] [production]
19:52 <jclark@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudvirtlocal1003'] [production]
19:52 <jclark@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cloudvirtlocal1002'] [production]
19:52 <jclark@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudvirtlocal1002'] [production]
19:51 <jclark@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cloudvirtlocal1001'] [production]
19:51 <jclark@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudvirtlocal1001'] [production]
19:48 <jclark@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirtlocal1001.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:35 <brett> Disable Puppet/PyBal on lvs3005 in preparation for reimaging - T321309 [production]
19:25 <mutante> mw2488 - scap pull - T334429 [production]