501-550 of 10000 results (83ms)
2023-11-22 §
02:01 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1158.eqiad.wmnet with OS bullseye [production]
01:27 <jclark@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['an-worker1158'] [production]
01:21 <jclark@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-worker1158'] [production]
00:55 <tstarling@deploy2002> Synchronized wmf-config/CommonSettings.php: enable LoginNotify seen subnets table g965663 T346989 (duration: 06m 23s) [production]
00:41 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host an-worker1158.eqiad.wmnet with OS bullseye [production]
2023-11-21 §
23:05 <vriley@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
23:05 <vriley@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: message - vriley@cumin1001" [production]
23:04 <vriley@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: message - vriley@cumin1001" [production]
23:02 <vriley@cumin1001> START - Cookbook sre.dns.netbox [production]
23:02 <vriley@cumin1001> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1036 [production]
23:01 <vriley@cumin1001> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1038 [production]
23:01 <vriley@cumin1001> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1037 [production]
23:00 <vriley@cumin1001> START - Cookbook sre.network.configure-switch-interfaces for host ganeti1038 [production]
23:00 <vriley@cumin1001> START - Cookbook sre.network.configure-switch-interfaces for host ganeti1037 [production]
23:00 <vriley@cumin1001> START - Cookbook sre.network.configure-switch-interfaces for host ganeti1036 [production]
22:59 <vriley@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:59 <vriley@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: message - vriley@cumin1001" [production]
22:58 <vriley@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: message - vriley@cumin1001" [production]
22:54 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for aqs1021.eqiad.wmnet [production]
22:54 <eevans@cumin1001> START - Cookbook sre.hosts.remove-downtime for aqs1021.eqiad.wmnet [production]
22:53 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs1021.eqiad.wmnet with OS bullseye [production]
22:48 <vriley@cumin1001> START - Cookbook sre.dns.netbox [production]
22:47 <vriley@cumin1001> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti1035 [production]
22:46 <vriley@cumin1001> START - Cookbook sre.network.configure-switch-interfaces for host ganeti1035 [production]
22:32 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1021.eqiad.wmnet with reason: host reimage [production]
22:30 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2179 (T348183)', diff saved to https://phabricator.wikimedia.org/P53687 and previous config saved to /var/cache/conftool/dbconfig/20231121-223053-arnaudb.json [production]
22:29 <eevans@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1021.eqiad.wmnet with reason: host reimage [production]
22:23 <vriley@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1038.mgmt.eqiad.wmnet with reboot policy FORCED [production]
22:20 <vriley@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1037.mgmt.eqiad.wmnet with reboot policy FORCED [production]
22:18 <vriley@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1036.mgmt.eqiad.wmnet with reboot policy FORCED [production]
22:18 <eevans@cumin1001> START - Cookbook sre.hosts.reimage for host aqs1021.eqiad.wmnet with OS bullseye [production]
22:17 <eevans@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host aqs1021.eqiad.wmnet with OS bullseye [production]
22:15 <vriley@cumin1001> START - Cookbook sre.hosts.provision for host ganeti1038.mgmt.eqiad.wmnet with reboot policy FORCED [production]
22:15 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P53686 and previous config saved to /var/cache/conftool/dbconfig/20231121-221547-arnaudb.json [production]
22:15 <vriley@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1035.mgmt.eqiad.wmnet with reboot policy FORCED [production]
22:12 <vriley@cumin1001> START - Cookbook sre.hosts.provision for host ganeti1037.mgmt.eqiad.wmnet with reboot policy FORCED [production]
22:11 <vriley@cumin1001> START - Cookbook sre.hosts.provision for host ganeti1036.mgmt.eqiad.wmnet with reboot policy FORCED [production]
22:08 <vriley@cumin1001> START - Cookbook sre.hosts.provision for host ganeti1035.mgmt.eqiad.wmnet with reboot policy FORCED [production]
22:06 <eevans@cumin1001> START - Cookbook sre.hosts.reimage for host aqs1021.eqiad.wmnet with OS bullseye [production]
22:02 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs1020.eqiad.wmnet with OS bullseye [production]
22:00 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P53685 and previous config saved to /var/cache/conftool/dbconfig/20231121-220040-arnaudb.json [production]
21:45 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2179 (T348183)', diff saved to https://phabricator.wikimedia.org/P53684 and previous config saved to /var/cache/conftool/dbconfig/20231121-214534-arnaudb.json [production]
21:40 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1020.eqiad.wmnet with reason: host reimage [production]
21:37 <eevans@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1020.eqiad.wmnet with reason: host reimage [production]
21:35 <catrope@deploy2002> Sync cancelled. [production]
21:31 <catrope@deploy2002> ssastry and catrope: Backport for [[gerrit:976330|ParserOutputPostCacheTransform: Don't reprocess content (T351461)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
21:29 <catrope@deploy2002> Started scap: Backport for [[gerrit:976330|ParserOutputPostCacheTransform: Don't reprocess content (T351461)]] [production]
21:29 <catrope@deploy2002> Finished scap: Backport for [[gerrit:976331|[parser] Broaden TOC placeholder regular expression]] (duration: 12m 40s) [production]
21:23 <catrope@deploy2002> catrope and ssastry: Continuing with sync [production]
21:18 <catrope@deploy2002> catrope and ssastry: Backport for [[gerrit:976331|[parser] Broaden TOC placeholder regular expression]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]