2101-2150 of 10000 results (77ms)
2023-01-13 §
12:38 <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on gitlab2002.wikimedia.org with reason: troubeleshoot backup restore on gitlab replica [production]
11:48 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "new bastions - jmm@cumin2002" [production]
11:45 <jmm@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "new bastions - jmm@cumin2002" [production]
10:53 <moritzm> installing bast5003 T324974 [production]
10:49 <jynus> restarting update-ubuntu-mirror on mirror1001 due to remote server concurrency issues [production]
09:41 <moritzm> installing bast4004 T324974 [production]
09:06 <moritzm> installing bast3006 T324974 [production]
02:33 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host druid1011.mgmt.eqiad.wmnet with reboot policy FORCED [production]
02:09 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host druid1011.mgmt.eqiad.wmnet with reboot policy FORCED [production]
02:08 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host druid1010.mgmt.eqiad.wmnet with reboot policy FORCED [production]
02:03 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host druid1009.mgmt.eqiad.wmnet with reboot policy FORCED [production]
01:55 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host druid1009.mgmt.eqiad.wmnet with reboot policy FORCED [production]
01:54 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host druid1009.mgmt.eqiad.wmnet with reboot policy FORCED [production]
01:53 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host druid1009.mgmt.eqiad.wmnet with reboot policy FORCED [production]
01:52 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host druid1009.mgmt.eqiad.wmnet with reboot policy FORCED [production]
01:36 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host druid1010.mgmt.eqiad.wmnet with reboot policy FORCED [production]
01:36 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host druid1009.mgmt.eqiad.wmnet with reboot policy FORCED [production]
01:26 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=False) upgrade firmware for hosts ['an-mariadb1002'] [production]
01:26 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-mariadb1002'] [production]
01:25 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=False) upgrade firmware for hosts ['an-mariadb1001'] [production]
01:25 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-mariadb1001'] [production]
01:21 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=True) upgrade firmware for hosts ['an-mariadb1002'] [production]
01:21 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=True) upgrade firmware for hosts ['an-mariadb1001'] [production]
01:04 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-mariadb1002'] [production]
01:04 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-mariadb1001'] [production]
01:03 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=False) upgrade firmware for hosts ['an-coord1004'] [production]
01:03 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-coord1004'] [production]
01:02 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=False) upgrade firmware for hosts ['an-coord1003'] [production]
01:02 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-coord1003'] [production]
00:54 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=True) upgrade firmware for hosts ['an-coord1004'] [production]
00:54 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=True) upgrade firmware for hosts ['an-coord1003'] [production]
00:41 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-coord1004'] [production]
00:40 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-coord1003'] [production]
00:36 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-mariadb1002.mgmt.eqiad.wmnet with reboot policy FORCED [production]
00:36 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-mariadb1001.mgmt.eqiad.wmnet with reboot policy FORCED [production]
00:16 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host an-mariadb1002.mgmt.eqiad.wmnet with reboot policy FORCED [production]
00:15 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host an-mariadb1001.mgmt.eqiad.wmnet with reboot policy FORCED [production]
00:15 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-coord1004.mgmt.eqiad.wmnet with reboot policy FORCED [production]
00:11 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-coord1003.mgmt.eqiad.wmnet with reboot policy FORCED [production]
2023-01-12 §
23:53 <zabe> start running cuc_comment_id population script on rest of sections in screens with --sleep 2 # T233004 [production]
23:50 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host an-coord1004.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:44 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host an-coord1003.mgmt.eqiad.wmnet with reboot policy FORCED [production]
23:13 <ebernhardson@deploy1002> Finished deploy [wikimedia/discovery/analytics@99a3e6f]: import_cirrus_index: use spark3 (duration: 02m 31s) [production]
23:10 <ebernhardson@deploy1002> Started deploy [wikimedia/discovery/analytics@99a3e6f]: import_cirrus_index: use spark3 [production]
23:08 <sbassett> Deployed (temporary) security mitigations for T326691 [production]
22:45 <mutante> people2002 - apt-get remove --purge rsync [production]
22:08 <zabe> start of "foreachwikiindblist s3.dblist extensions/CheckUser/maintenance/populateCucComment.php" in a screen in mwmaint1002 # T233004 [production]
22:07 <thcipriani> end UTC late backport [production]
22:06 <thcipriani@deploy1002> Finished scap: Backport for [[gerrit:879161|cirrus: Divert requests with x-public-cloud set to a dedicated pool counter (T326757)]], [[gerrit:862343|cirrus: Disable incoming link counting (T317023)]] (duration: 09m 23s) [production]
21:59 <krinkle@deploy1002> Finished deploy [performance/navtiming@172cc22]: (no justification provided) (duration: 00m 08s) [production]