301-350 of 10000 results (57ms)
2023-01-30 ยง
20:26 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2033.codfw.wmnet with OS bullseye [production]
20:23 <zabe@deploy1002> Finished scap: Backport for [[gerrit:885002|slwiki: Raise AF emergency disable treshold+count (T328366)]] (duration: 07m 32s) [production]
20:17 <zabe@deploy1002> zabe: Backport for [[gerrit:885002|slwiki: Raise AF emergency disable treshold+count (T328366)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
20:16 <zabe@deploy1002> Started scap: Backport for [[gerrit:885002|slwiki: Raise AF emergency disable treshold+count (T328366)]] [production]
20:15 <brett@cumin1001> START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye [production]
20:14 <brett@cumin1001> conftool action : set/pooled=yes; selector: name=cp4044.ulsfo.wmnet [production]
20:12 <brett@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4044.ulsfo.wmnet with OS bullseye [production]
20:06 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2033.codfw.wmnet with reason: host reimage [production]
20:03 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp2033.codfw.wmnet with reason: host reimage [production]
19:57 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp3052.esams.wmnet,service=ats-be [production]
19:57 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp3052.esams.wmnet,service=cdn [production]
19:50 <brett@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4044.ulsfo.wmnet with reason: host reimage [production]
19:48 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3052.esams.wmnet with OS bullseye [production]
19:47 <brett@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp4044.ulsfo.wmnet with reason: host reimage [production]
19:44 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp2033.codfw.wmnet with OS bullseye [production]
19:36 <bking@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
19:35 <bking@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
19:26 <brett@cumin1001> START - Cookbook sre.hosts.reimage for host cp4044.ulsfo.wmnet with OS bullseye [production]
19:26 <brett@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4044.ulsfo.wmnet with OS bullseye [production]
19:25 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3052.esams.wmnet with reason: host reimage [production]
19:22 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp3052.esams.wmnet with reason: host reimage [production]
19:21 <cstone> payments-wiki upgraded from 653c7cc8 to f20a2208 [production]
19:16 <brett@cumin1001> START - Cookbook sre.hosts.reimage for host cp4044.ulsfo.wmnet with OS bullseye [production]
19:15 <brett@cumin1001> conftool action : set/pooled=yes; selector: name=cp4051.ulsfo.wmnet [production]
19:01 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp3052.esams.wmnet with OS bullseye [production]
18:46 <sukhe@cumin2002> END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['cp3052.esams.wmnet'] [production]
18:46 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp3052.esams.wmnet'] [production]
18:46 <sukhe@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp3052.esams.wmnet'] [production]
18:45 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp3052.esams.wmnet'] [production]
18:45 <sukhe@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp3052.esams.wmnet with OS bullseye [production]
18:38 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp3052.esams.wmnet with OS bullseye [production]
18:37 <sukhe@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp3052.esams.wmnet with OS bullseye [production]
18:37 <sukhe@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp3052.esams.wmnet'] [production]
18:37 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp3052.esams.wmnet'] [production]
18:34 <brett@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4051.ulsfo.wmnet with OS bullseye [production]
18:29 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on vrts2001.codfw.wmnet with reason: installation failed due to read-only database [production]
18:29 <aokoth@cumin1001> START - Cookbook sre.hosts.downtime for 14 days, 0:00:00 on vrts2001.codfw.wmnet with reason: installation failed due to read-only database [production]
18:19 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp3052.esams.wmnet with OS bullseye [production]
18:19 <sukhe@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp3052.esams.wmnet with OS bullseye [production]
18:10 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp3052.esams.wmnet with OS bullseye [production]
18:08 <sukhe@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp3052.esams.wmnet [production]
18:07 <brett@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4051.ulsfo.wmnet with reason: host reimage [production]
18:04 <brett@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp4051.ulsfo.wmnet with reason: host reimage [production]
18:01 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:884427|[Growth] Remove wgGERecentChangesUnstarredMenteesFilterEnabled]] (duration: 07m 59s) [production]
17:53 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:884427|[Growth] Remove wgGERecentChangesUnstarredMenteesFilterEnabled]] [production]
17:49 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2177 (T328255)', diff saved to https://phabricator.wikimedia.org/P43517 and previous config saved to /var/cache/conftool/dbconfig/20230130-174957-ladsgroup.json [production]
17:49 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp3052.esams.wmnet [production]
17:43 <brett@cumin1001> START - Cookbook sre.hosts.reimage for host cp4051.ulsfo.wmnet with OS bullseye [production]
17:43 <brett@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4051.ulsfo.wmnet with OS bullseye [production]
17:36 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp5026.eqsin.wmnet,service=ats-be [production]