2001-2050 of 10000 results (91ms)
2024-04-11 ยง
12:53 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db[2132,2160].codfw.wmnet with reason: reboot [production]
12:52 <jayme@deploy1002> helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
12:52 <jayme@deploy1002> helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
12:51 <jayme@deploy1002> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [production]
12:50 <jayme@deploy1002> helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [production]
12:49 <jayme@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
12:49 <jayme@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
12:45 <ayounsi@cumin1002> START - Cookbook sre.dns.netbox [production]
12:24 <btullis@deploy1002> helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply [production]
12:24 <btullis@deploy1002> helmfile [eqiad] START helmfile.d/services/editor-analytics: apply [production]
12:23 <btullis@deploy1002> helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply [production]
12:22 <btullis@deploy1002> helmfile [codfw] START helmfile.d/services/editor-analytics: apply [production]
12:21 <ayounsi@cumin1002> START - Cookbook sre.hosts.reimage for host testvm2008.wikimedia.org with OS bookworm [production]
12:21 <ayounsi@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2008.wikimedia.org - ayounsi@cumin1002" [production]
12:20 <btullis@deploy1002> helmfile [staging] DONE helmfile.d/services/editor-analytics: apply [production]
12:20 <ayounsi@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2008.wikimedia.org - ayounsi@cumin1002" [production]
12:20 <ayounsi@cumin1002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2008.wikimedia.org on all recursors [production]
12:20 <btullis@deploy1002> helmfile [staging] START helmfile.d/services/editor-analytics: apply [production]
12:19 <ayounsi@cumin1002> START - Cookbook sre.dns.wipe-cache testvm2008.wikimedia.org on all recursors [production]
12:19 <ayounsi@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
12:19 <ayounsi@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2008.wikimedia.org - ayounsi@cumin1002" [production]
12:18 <ayounsi@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2008.wikimedia.org - ayounsi@cumin1002" [production]
12:16 <hnowlan@deploy1002> helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
12:16 <ayounsi@cumin1002> START - Cookbook sre.dns.netbox [production]
12:16 <ayounsi@cumin1002> START - Cookbook sre.ganeti.makevm for new host testvm2008.wikimedia.org [production]
12:16 <btullis@cumin1002> START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm [production]
12:16 <ayounsi@cumin1002> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host testvm2008.wikimedia.org [production]
12:16 <ayounsi@cumin1002> END (FAIL) - Cookbook sre.dns.netbox (exit_code=97) [production]
12:16 <ayounsi@cumin1002> END (ERROR) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=97) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2008.wikimedia.org - ayounsi@cumin1002" [production]
12:16 <ayounsi@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2008.wikimedia.org - ayounsi@cumin1002" [production]
12:15 <hnowlan@deploy1002> helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply [production]
12:15 <moritzm> installing gnutls28 security updates [production]
12:14 <hnowlan@deploy1002> helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
12:13 <ayounsi@cumin1002> START - Cookbook sre.dns.netbox [production]
12:13 <ayounsi@cumin1002> START - Cookbook sre.ganeti.makevm for new host testvm2008.wikimedia.org [production]
12:13 <hnowlan@deploy1002> helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply [production]
12:13 <ayounsi@cumin1002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2008.wikimedia.org [production]
12:13 <ayounsi@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
12:13 <ayounsi@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2008.wikimedia.org decommissioned, removing all IPs except the asset tag one - ayounsi@cumin1002" [production]
12:13 <btullis@cumin1002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host matomo1003.eqiad.wmnet with OS bullseye [production]
12:12 <ayounsi@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2008.wikimedia.org decommissioned, removing all IPs except the asset tag one - ayounsi@cumin1002" [production]
12:10 <ayounsi@cumin1002> START - Cookbook sre.dns.netbox [production]
12:06 <ayounsi@cumin1002> START - Cookbook sre.hosts.decommission for hosts testvm2008.wikimedia.org [production]
12:06 <ayounsi@cumin1002> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=97) for new host testvm2008.wikimedia.org [production]
12:05 <ayounsi@cumin1002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host testvm2008.wikimedia.org with OS bookworm [production]
12:02 <cgoubert@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply [production]
12:01 <cgoubert@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply [production]
12:01 <cgoubert@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply [production]
11:59 <cgoubert@deploy1002> helmfile [codfw] START helmfile.d/services/mw-api-ext: apply [production]
11:59 <cgoubert@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-web: apply [production]