2551-2600 of 10000 results (94ms)
2023-11-27 ยง
22:17 <jhathaway@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhathaway@cumin1001" [production]
22:13 <jhathaway@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhathaway@cumin1001" [production]
21:59 <jhathaway@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1157.eqiad.wmnet with reason: host reimage [production]
21:56 <jhathaway@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1157.eqiad.wmnet with reason: host reimage [production]
21:42 <jhathaway@cumin1001> START - Cookbook sre.hosts.reimage for host an-worker1157.eqiad.wmnet with OS bullseye [production]
21:28 <cjming> end of UTC late backport window [production]
21:26 <cjming@deploy2002> Finished scap: Backport for [[gerrit:976161|ORES: Set default value of OresLiftWingAddHostHeader to true (T351703)]] (duration: 07m 57s) [production]
21:19 <cjming@deploy2002> isaranto and cjming: Continuing with sync [production]
21:19 <cjming@deploy2002> isaranto and cjming: Backport for [[gerrit:976161|ORES: Set default value of OresLiftWingAddHostHeader to true (T351703)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
21:18 <cjming@deploy2002> Started scap: Backport for [[gerrit:976161|ORES: Set default value of OresLiftWingAddHostHeader to true (T351703)]] [production]
21:15 <cjming@deploy2002> Finished scap: Backport for [[gerrit:976864|CentralAuth: Fix wikisource.org cookie handling (T351685)]] (duration: 11m 26s) [production]
21:09 <cjming@deploy2002> cjming and tgr: Continuing with sync [production]
21:07 <pt1979@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-worker1157.eqiad.wmnet with OS bullseye [production]
21:05 <cjming@deploy2002> cjming and tgr: Backport for [[gerrit:976864|CentralAuth: Fix wikisource.org cookie handling (T351685)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
21:04 <cjming@deploy2002> Started scap: Backport for [[gerrit:976864|CentralAuth: Fix wikisource.org cookie handling (T351685)]] [production]
21:02 <btullis@deploy2002> Finished deploy [airflow-dags/analytics_test@0283c11]: (no justification provided) (duration: 00m 11s) [production]
21:02 <btullis@deploy2002> Started deploy [airflow-dags/analytics_test@0283c11]: (no justification provided) [production]
20:52 <pt1979@cumin1001> START - Cookbook sre.hosts.reimage for host an-worker1157.eqiad.wmnet with OS bullseye [production]
20:16 <vgutierrez> rolling restart of pybal on lvs4010 and lvs4008 - T351069 [production]
19:53 <vgutierrez> restarting pybal on lvs4008 (effectively enabling IPIP encapsulation on ncredir@ulsfo) - T351069 [production]
19:50 <vgutierrez> restarting pybal on lvs4010 - T351069 [production]
19:21 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on planet1003.eqiad.wmnet with reason: maintenance [production]
19:21 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on planet1003.eqiad.wmnet with reason: maintenance [production]
19:20 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on planet2003.codfw.wmnet with reason: maintenance [production]
19:20 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on planet2003.codfw.wmnet with reason: maintenance [production]
18:56 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirtlocal1003.eqiad.wmnet with OS bookworm [production]
18:37 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host planet1003.eqiad.wmnet with OS bookworm [production]
18:28 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirtlocal1003.eqiad.wmnet with reason: host reimage [production]
18:25 <andrew@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirtlocal1003.eqiad.wmnet with reason: host reimage [production]
18:25 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on planet1003.eqiad.wmnet with reason: host reimage [production]
18:22 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on planet1003.eqiad.wmnet with reason: host reimage [production]
18:11 <dzahn@cumin1001> START - Cookbook sre.hosts.reimage for host planet1003.eqiad.wmnet with OS bookworm [production]
18:09 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirtlocal1003.eqiad.wmnet with OS bookworm [production]
18:07 <vgutierrez> restarting pybal on lvs4010 - T351069 [production]
17:52 <vgutierrez> upload ipip-multiqueue-optimizer 0.2 to apt.wm.o (bullseye) - T351069 [production]
17:48 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirtlocal1002.eqiad.wmnet with OS bookworm [production]
17:25 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirtlocal1002.eqiad.wmnet with reason: host reimage [production]
17:22 <andrew@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirtlocal1002.eqiad.wmnet with reason: host reimage [production]
17:13 <jhancock@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ganeti2033'] [production]
17:07 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti2033'] [production]
17:07 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['ganeti2033'] [production]
17:07 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['kubernetes2058'] [production]
17:07 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['kubernetes2057'] [production]
17:07 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti2033'] [production]
17:06 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['kubernetes2058'] [production]
17:06 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['kubernetes2057'] [production]
17:06 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirtlocal1002.eqiad.wmnet with OS bookworm [production]
16:56 <jdrewniak@deploy2002> Synchronized portals: Wikimedia Portals Update: [[gerrit:977735| Bumping portals to master (T128546)]] (duration: 06m 31s) [production]
16:52 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirtlocal1001.eqiad.wmnet with OS bookworm [production]
16:50 <jbond@cumin1001> END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: dbbackups::monitoring [production]