1401-1450 of 10000 results (97ms)
2024-06-27 §
16:05 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1024.eqiad.wmnet with OS bullseye [production]
16:03 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1022.eqiad.wmnet with OS bullseye [production]
16:01 <andrew@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirtlocal1002.eqiad.wmnet with reason: host reimage [production]
16:00 <akosiaris@deploy1002> helmfile [eqiad] DONE helmfile.d/services/citoid: sync [production]
16:00 <akosiaris@deploy1002> helmfile [eqiad] START helmfile.d/services/citoid: sync [production]
15:58 <andrew@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirtlocal1002.eqiad.wmnet with reason: host reimage [production]
15:56 <brett@cumin2002> START - Cookbook sre.hosts.reimage for host cp5021.eqsin.wmnet with OS bullseye [production]
15:56 <brett@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5021.eqsin.wmnet with OS bullseye [production]
15:56 <jhancock@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dbproxy2005 [production]
15:56 <jhancock@cumin2002> START - Cookbook sre.network.configure-switch-interfaces for host dbproxy2005 [production]
15:55 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1026.eqiad.wmnet with reason: host reimage [production]
15:51 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1025.eqiad.wmnet with reason: host reimage [production]
15:51 <arnaudb@cumin1002> dbctl commit (dc=all): 'es1037 (re)pooling @ 25%: post T365988 repool', diff saved to https://phabricator.wikimedia.org/P65534 and previous config saved to /var/cache/conftool/dbconfig/20240627-155118-arnaudb.json [production]
15:48 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1023.eqiad.wmnet with reason: host reimage [production]
15:46 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1024.eqiad.wmnet with reason: host reimage [production]
15:44 <cgoubert@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1025.eqiad.wmnet with reason: host reimage [production]
15:43 <hnowlan> restarted ferm on 8 failing k8s workers [production]
15:43 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1022.eqiad.wmnet with reason: host reimage [production]
15:42 <cgoubert@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1026.eqiad.wmnet with reason: host reimage [production]
15:42 <cgoubert@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1024.eqiad.wmnet with reason: host reimage [production]
15:41 <andrew@cumin1002> START - Cookbook sre.hosts.reimage for host cloudvirtlocal1002.eqiad.wmnet with OS bookworm [production]
15:41 <cgoubert@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1023.eqiad.wmnet with reason: host reimage [production]
15:40 <cgoubert@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1022.eqiad.wmnet with reason: host reimage [production]
15:36 <arnaudb@cumin1002> dbctl commit (dc=all): 'es1037 (re)pooling @ 10%: post T365988 repool', diff saved to https://phabricator.wikimedia.org/P65533 and previous config saved to /var/cache/conftool/dbconfig/20240627-153613-arnaudb.json [production]
15:29 <cgoubert@cumin1002> START - Cookbook sre.hosts.reimage for host wikikube-worker1025.eqiad.wmnet with OS bullseye [production]
15:29 <cgoubert@cumin1002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1025.eqiad.wmnet on all recursors [production]
15:29 <cgoubert@cumin1002> START - Cookbook sre.dns.wipe-cache wikikube-worker1025.eqiad.wmnet on all recursors [production]
15:28 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1373 to wikikube-worker1025 [production]
15:28 <cgoubert@cumin1002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1025 [production]
15:28 <cgoubert@cumin1002> START - Cookbook sre.hosts.reimage for host wikikube-worker1026.eqiad.wmnet with OS bullseye [production]
15:28 <cgoubert@cumin1002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1026.eqiad.wmnet on all recursors [production]
15:28 <cgoubert@cumin1002> START - Cookbook sre.dns.wipe-cache wikikube-worker1026.eqiad.wmnet on all recursors [production]
15:27 <cgoubert@cumin1002> START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1025 [production]
15:27 <cgoubert@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:27 <cgoubert@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1373 to wikikube-worker1025 - cgoubert@cumin1002" [production]
15:27 <cgoubert@cumin1002> START - Cookbook sre.hosts.reimage for host wikikube-worker1024.eqiad.wmnet with OS bullseye [production]
15:27 <cgoubert@cumin1002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1024.eqiad.wmnet on all recursors [production]
15:27 <cgoubert@cumin1002> START - Cookbook sre.dns.wipe-cache wikikube-worker1024.eqiad.wmnet on all recursors [production]
15:26 <cgoubert@cumin1002> START - Cookbook sre.hosts.reimage for host wikikube-worker1023.eqiad.wmnet with OS bullseye [production]
15:26 <cgoubert@cumin1002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1023.eqiad.wmnet on all recursors [production]
15:26 <cgoubert@cumin1002> START - Cookbook sre.dns.wipe-cache wikikube-worker1023.eqiad.wmnet on all recursors [production]
15:25 <cgoubert@cumin1002> START - Cookbook sre.hosts.reimage for host wikikube-worker1022.eqiad.wmnet with OS bullseye [production]
15:25 <cgoubert@cumin1002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1022.eqiad.wmnet on all recursors [production]
15:25 <cgoubert@cumin1002> START - Cookbook sre.dns.wipe-cache wikikube-worker1022.eqiad.wmnet on all recursors [production]
15:25 <pmiazga> T367901 mwmaint1002: Ran `mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=rowiki --logwiki=metawiki 'Rui_Filipe_Fernandes' '44_Gabriel’` [production]
15:24 <cgoubert@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1373 to wikikube-worker1025 - cgoubert@cumin1002" [production]
15:23 <dcaro@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd1007.eqiad.wmnet [production]
15:21 <cgoubert@cumin1002> START - Cookbook sre.dns.netbox [production]
15:21 <cgoubert@cumin1002> START - Cookbook sre.hosts.rename from mw1373 to wikikube-worker1025 [production]
15:21 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1366 to wikikube-worker1024 [production]