651-700 of 10000 results (25ms)
2024-10-03 ยง
14:52 <xcollazo@deploy2002> Started deploy [airflow-dags/analytics@b715af7]: Deploy latest DAGs to the analytics Airflow instance. T373694. T375402 [production]
14:52 <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [admin]
14:51 <arturo> delete default security group rule list, now tracking them via tofu-infra [admin]
14:48 <aborrero@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [admin]
14:47 <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [admin]
14:31 <jclark@cumin1002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host aqs1022.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
14:31 <jclark@cumin1002> START - Cookbook sre.hosts.provision for host aqs1022.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
14:30 <jclark@cumin1002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host aqs1022 [production]
14:29 <jclark@cumin1002> START - Cookbook sre.network.configure-switch-interfaces for host aqs1022 [production]
14:29 <jclark@cumin1002> END (ERROR) - Cookbook sre.network.configure-switch-interfaces (exit_code=97) for host aqs1022 [production]
14:28 <jclark@cumin1002> START - Cookbook sre.network.configure-switch-interfaces for host aqs1022 [production]
14:28 <jclark@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:28 <jclark@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt aqs1022 - jclark@cumin1002" [production]
14:26 <jclark@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt aqs1022 - jclark@cumin1002" [production]
14:23 <jclark@cumin1002> START - Cookbook sre.dns.netbox [production]
14:04 <dcaro> deploying tekton upgrade (builds-builder + builds-api https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531) T374908 [toolsbeta]
14:03 <dcaro> deploying tekton upgrade (builds-builder + builds-api https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531) [toolsbeta]
13:55 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
13:54 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
13:46 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc2004.wikimedia.org [production]
13:42 <elukey@cumin1002> START - Cookbook sre.hosts.reboot-single for host irc2004.wikimedia.org [production]
13:40 <elukey@cumin1002> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host irc2004.wikimedia.org [production]
13:40 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host irc2004.wikimedia.org with OS bookworm [production]
13:32 <stevemunene@cumin1002> START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye [production]
13:31 <stevemunene@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1176.eqiad.wmnet with OS bullseye [production]
13:30 <stevemunene@cumin1002> START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bullseye [production]
13:26 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on irc2004.wikimedia.org with reason: host reimage [production]
13:23 <elukey@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on irc2004.wikimedia.org with reason: host reimage [production]
13:22 <wmbot~lucaswerkmeister@tools-bastion-13> deployed a81b469204 (l10n updates: ms-arab) [tools.lexeme-forms]
13:10 <elukey@cumin1002> START - Cookbook sre.hosts.reimage for host irc2004.wikimedia.org with OS bookworm [production]
13:09 <elukey@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM irc2004.wikimedia.org - elukey@cumin1002" [production]
13:09 <elukey@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM irc2004.wikimedia.org - elukey@cumin1002" [production]
13:09 <elukey@cumin1002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) irc2004.wikimedia.org on all recursors [production]
13:09 <elukey@cumin1002> START - Cookbook sre.dns.wipe-cache irc2004.wikimedia.org on all recursors [production]
13:09 <elukey@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:09 <elukey@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM irc2004.wikimedia.org - elukey@cumin1002" [production]
13:08 <elukey@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM irc2004.wikimedia.org - elukey@cumin1002" [production]
13:00 <elukey@cumin1002> START - Cookbook sre.dns.netbox [production]
13:00 <elukey@cumin1002> START - Cookbook sre.ganeti.makevm for new host irc2004.wikimedia.org [production]
12:20 <urbanecm@deploy2002> Finished scap sync-world: Backport for [[gerrit:1077673|ReassignMenteesJob: Do not schedule follow-up jobs when first job fails (T376124)]] (duration: 06m 47s) [production]
12:13 <urbanecm@deploy2002> Started scap sync-world: Backport for [[gerrit:1077673|ReassignMenteesJob: Do not schedule follow-up jobs when first job fails (T376124)]] [production]
12:13 <urbanecm@deploy2002> scap failed: <UnboundLocalError> local variable 'e' referenced before assignment (scap version: 4.108.0-1) (duration: 08m 02s) [production]
12:13 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host logging-hd2004.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]
12:09 <elukey@cumin1002> START - Cookbook sre.hosts.provision for host logging-hd2004.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]
12:05 <urbanecm@deploy2002> Started scap sync-world: Backport for [[gerrit:1077673|ReassignMenteesJob: Do not schedule follow-up jobs when first job fails (T376124)]] [production]
12:05 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host logging-hd2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]
12:02 <elukey@cumin1002> START - Cookbook sre.hosts.provision for host logging-hd2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]
11:15 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1178 (T367856)', diff saved to https://phabricator.wikimedia.org/P69458 and previous config saved to /var/cache/conftool/dbconfig/20241003-111544-ladsgroup.json [production]
11:15 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 12:00:00 on db1178.eqiad.wmnet with reason: Maintenance [production]
11:15 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 2 days, 12:00:00 on db1178.eqiad.wmnet with reason: Maintenance [production]