2651-2700 of 10000 results (92ms)
2023-08-29 ยง
18:56 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudservices1006.eqiad.wmnet with reason: host reimage [production]
18:55 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host cloudservices1006.eqiad.wmnet with OS bullseye [production]
18:55 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudservices1006.eqiad.wmnet with OS bullseye [production]
18:54 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host cloudservices1006.eqiad.wmnet with OS bullseye [production]
18:52 <eevans@cumin1001> START - Cookbook sre.hosts.reboot-single for host restbase1025.eqiad.wmnet [production]
18:37 <aokoth@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host vrts1002.eqiad.wmnet [production]
18:37 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host vrts1002.eqiad.wmnet with OS bullseye [production]
18:32 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase1032.eqiad.wmnet [production]
18:24 <eevans@cumin1001> START - Cookbook sre.hosts.reboot-single for host restbase1032.eqiad.wmnet [production]
18:24 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on vrts1002.eqiad.wmnet with reason: host reimage [production]
18:22 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2155 (T343718)', diff saved to https://phabricator.wikimedia.org/P51946 and previous config saved to /var/cache/conftool/dbconfig/20230829-182251-ladsgroup.json [production]
18:22 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
18:22 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
18:22 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance [production]
18:22 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance [production]
18:22 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2147 (T343718)', diff saved to https://phabricator.wikimedia.org/P51945 and previous config saved to /var/cache/conftool/dbconfig/20230829-182225-ladsgroup.json [production]
18:21 <aokoth@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on vrts1002.eqiad.wmnet with reason: host reimage [production]
18:16 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase1029.eqiad.wmnet [production]
18:14 <jhuneidi@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.41.0-wmf.24 refs T343726 [production]
18:12 <aokoth@cumin1001> START - Cookbook sre.hosts.reimage for host vrts1002.eqiad.wmnet with OS bullseye [production]
18:10 <aokoth@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM vrts1002.eqiad.wmnet - aokoth@cumin1001" [production]
18:09 <aokoth@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM vrts1002.eqiad.wmnet - aokoth@cumin1001" [production]
18:08 <aokoth@cumin1001> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) vrts1002.eqiad.wmnet on all recursors [production]
18:08 <aokoth@cumin1001> START - Cookbook sre.dns.wipe-cache vrts1002.eqiad.wmnet on all recursors [production]
18:08 <aokoth@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:08 <aokoth@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM vrts1002.eqiad.wmnet - aokoth@cumin1001" [production]
18:08 <aokoth@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM vrts1002.eqiad.wmnet - aokoth@cumin1001" [production]
18:07 <eevans@cumin1001> START - Cookbook sre.hosts.reboot-single for host restbase1029.eqiad.wmnet [production]
18:07 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P51944 and previous config saved to /var/cache/conftool/dbconfig/20230829-180719-ladsgroup.json [production]
18:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1160 (T343718)', diff saved to https://phabricator.wikimedia.org/P51943 and previous config saved to /var/cache/conftool/dbconfig/20230829-180613-ladsgroup.json [production]
18:06 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1160.eqiad.wmnet with reason: Maintenance [production]
18:05 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1160.eqiad.wmnet with reason: Maintenance [production]
18:04 <aokoth@cumin1001> START - Cookbook sre.dns.netbox [production]
18:04 <aokoth@cumin1001> START - Cookbook sre.ganeti.makevm for new host vrts1002.eqiad.wmnet [production]
18:00 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:953314|Growth: Disable Add an image on all wikis (T345188)]] (duration: 06m 47s) [production]
17:59 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase1024.eqiad.wmnet [production]
17:55 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudservices1006.eqiad.wmnet with OS bullseye [production]
17:53 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:953314|Growth: Disable Add an image on all wikis (T345188)]] [production]
17:52 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P51942 and previous config saved to /var/cache/conftool/dbconfig/20230829-175213-ladsgroup.json [production]
17:51 <jhuneidi@deploy1002> Pruned MediaWiki: 1.41.0-wmf.22 (duration: 02m 11s) [production]
17:49 <eevans@cumin1001> START - Cookbook sre.hosts.reboot-single for host restbase1024.eqiad.wmnet [production]
17:48 <jhuneidi@deploy1002> Finished scap: testwikis wikis to 1.41.0-wmf.24 refs T343726 (duration: 43m 27s) [production]
17:41 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase1023.eqiad.wmnet [production]
17:37 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2147 (T343718)', diff saved to https://phabricator.wikimedia.org/P51941 and previous config saved to /var/cache/conftool/dbconfig/20230829-173707-ladsgroup.json [production]
17:33 <eevans@cumin1001> START - Cookbook sre.hosts.reboot-single for host restbase1023.eqiad.wmnet [production]
17:05 <jhuneidi@deploy1002> Started scap: testwikis wikis to 1.41.0-wmf.24 refs T343726 [production]
16:31 <akosiaris@deploy1002> helmfile [codfw] DONE helmfile.d/services/cxserver: apply [production]
16:30 <akosiaris@deploy1002> helmfile [codfw] START helmfile.d/services/cxserver: apply [production]
16:30 <akosiaris@deploy1002> helmfile [eqiad] DONE helmfile.d/services/cxserver: apply [production]
16:30 <akosiaris@deploy1002> helmfile [eqiad] START helmfile.d/services/cxserver: apply [production]