1501-1550 of 10000 results (97ms)
2023-11-28 §
01:09 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host elastic2096.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:09 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti2034'] [production]
01:09 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['ganeti2034'] [production]
01:08 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti2034'] [production]
01:07 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti2034.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:07 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['elastic2101'] [production]
01:07 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['elastic2101'] [production]
01:07 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['elastic2101'] [production]
01:06 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['elastic2097'] [production]
01:06 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host elastic2101.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:06 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['elastic2097'] [production]
01:05 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['elastic2097'] [production]
01:04 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host elastic2097.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:58 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host elastic2096.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:57 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host elastic2096.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:57 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host ganeti2034.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:56 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host elastic2101.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:56 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host elastic2097.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:56 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host elastic2096.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:52 <pt1979@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host restbase2029.codfw.wmnet with OS bullseye [production]
00:38 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host restbase2028.codfw.wmnet with OS bullseye [production]
00:38 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
00:36 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
00:33 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host restbase2029.codfw.wmnet with OS bullseye [production]
00:19 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on restbase2028.codfw.wmnet with reason: host reimage [production]
00:15 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on restbase2028.codfw.wmnet with reason: host reimage [production]
2023-11-27 §
23:47 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host restbase2028.codfw.wmnet with OS bullseye [production]
22:55 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on planet1002.eqiad.wmnet with reason: maintenance [production]
22:54 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on planet1002.eqiad.wmnet with reason: maintenance [production]
22:17 <jhathaway@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1157.eqiad.wmnet with OS bullseye [production]
22:17 <jhathaway@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhathaway@cumin1001" [production]
22:13 <jhathaway@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhathaway@cumin1001" [production]
21:59 <jhathaway@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1157.eqiad.wmnet with reason: host reimage [production]
21:56 <jhathaway@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1157.eqiad.wmnet with reason: host reimage [production]
21:42 <jhathaway@cumin1001> START - Cookbook sre.hosts.reimage for host an-worker1157.eqiad.wmnet with OS bullseye [production]
21:28 <cjming> end of UTC late backport window [production]
21:26 <cjming@deploy2002> Finished scap: Backport for [[gerrit:976161|ORES: Set default value of OresLiftWingAddHostHeader to true (T351703)]] (duration: 07m 57s) [production]
21:19 <cjming@deploy2002> isaranto and cjming: Continuing with sync [production]
21:19 <cjming@deploy2002> isaranto and cjming: Backport for [[gerrit:976161|ORES: Set default value of OresLiftWingAddHostHeader to true (T351703)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
21:18 <cjming@deploy2002> Started scap: Backport for [[gerrit:976161|ORES: Set default value of OresLiftWingAddHostHeader to true (T351703)]] [production]
21:15 <cjming@deploy2002> Finished scap: Backport for [[gerrit:976864|CentralAuth: Fix wikisource.org cookie handling (T351685)]] (duration: 11m 26s) [production]
21:09 <cjming@deploy2002> cjming and tgr: Continuing with sync [production]
21:07 <pt1979@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-worker1157.eqiad.wmnet with OS bullseye [production]
21:05 <cjming@deploy2002> cjming and tgr: Backport for [[gerrit:976864|CentralAuth: Fix wikisource.org cookie handling (T351685)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
21:04 <cjming@deploy2002> Started scap: Backport for [[gerrit:976864|CentralAuth: Fix wikisource.org cookie handling (T351685)]] [production]
21:02 <btullis@deploy2002> Finished deploy [airflow-dags/analytics_test@0283c11]: (no justification provided) (duration: 00m 11s) [production]
21:02 <btullis@deploy2002> Started deploy [airflow-dags/analytics_test@0283c11]: (no justification provided) [production]
20:52 <pt1979@cumin1001> START - Cookbook sre.hosts.reimage for host an-worker1157.eqiad.wmnet with OS bullseye [production]
20:16 <vgutierrez> rolling restart of pybal on lvs4010 and lvs4008 - T351069 [production]
19:53 <vgutierrez> restarting pybal on lvs4008 (effectively enabling IPIP encapsulation on ncredir@ulsfo) - T351069 [production]