4251-4300 of 10000 results (82ms)
2023-01-30 ยง
21:25 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp2030.codfw.wmnet with OS bullseye [production]
21:24 <eevans@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching restbase2020.codfw.wmnet: Replace Cassandra keys & certs - eevans@cumin1001 [production]
21:23 <urbanecm@deploy1002> tgr and urbanecm and jdlrobson and legoktm: Backport for [[gerrit:883301|Enable WelcomeSurvey at viwiki (T325376)]], [[gerrit:884930|Fix grid blowout with limited width turned off (T327423)]], [[gerrit:884929|Support new style of table of contents (T327942)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
21:21 <eevans@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase2019.codfw.wmnet: Replace Cassandra keys & certs - eevans@cumin1001 [production]
21:21 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:883301|Enable WelcomeSurvey at viwiki (T325376)]], [[gerrit:884930|Fix grid blowout with limited width turned off (T327423)]], [[gerrit:884929|Support new style of table of contents (T327942)]] [production]
21:21 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:884474|InitialiseSettings: add zhwiki to wgPageAssessmentsSubprojects (T326387)]] (duration: 19m 51s) [production]
21:11 <eevans@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching restbase2019.codfw.wmnet: Replace Cassandra keys & certs - eevans@cumin1001 [production]
21:03 <urbanecm@deploy1002> urbanecm and musikanimal: Backport for [[gerrit:884474|InitialiseSettings: add zhwiki to wgPageAssessmentsSubprojects (T326387)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
21:02 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp2033.codfw.wmnet,service=ats-be [production]
21:02 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp2033.codfw.wmnet,service=cdn [production]
21:01 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:884474|InitialiseSettings: add zhwiki to wgPageAssessmentsSubprojects (T326387)]] [production]
20:59 <brett@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage [production]
20:56 <brett@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage [production]
20:51 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
20:50 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
20:35 <brett@cumin1001> START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye [production]
20:35 <brett@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4052.ulsfo.wmnet with OS bullseye [production]
20:26 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2033.codfw.wmnet with OS bullseye [production]
20:23 <zabe@deploy1002> Finished scap: Backport for [[gerrit:885002|slwiki: Raise AF emergency disable treshold+count (T328366)]] (duration: 07m 32s) [production]
20:17 <zabe@deploy1002> zabe: Backport for [[gerrit:885002|slwiki: Raise AF emergency disable treshold+count (T328366)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
20:16 <zabe@deploy1002> Started scap: Backport for [[gerrit:885002|slwiki: Raise AF emergency disable treshold+count (T328366)]] [production]
20:15 <brett@cumin1001> START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye [production]
20:14 <brett@cumin1001> conftool action : set/pooled=yes; selector: name=cp4044.ulsfo.wmnet [production]
20:12 <brett@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4044.ulsfo.wmnet with OS bullseye [production]
20:06 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2033.codfw.wmnet with reason: host reimage [production]
20:03 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp2033.codfw.wmnet with reason: host reimage [production]
19:57 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp3052.esams.wmnet,service=ats-be [production]
19:57 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp3052.esams.wmnet,service=cdn [production]
19:50 <brett@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4044.ulsfo.wmnet with reason: host reimage [production]
19:48 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3052.esams.wmnet with OS bullseye [production]
19:47 <brett@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp4044.ulsfo.wmnet with reason: host reimage [production]
19:44 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp2033.codfw.wmnet with OS bullseye [production]
19:36 <bking@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
19:35 <bking@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
19:26 <brett@cumin1001> START - Cookbook sre.hosts.reimage for host cp4044.ulsfo.wmnet with OS bullseye [production]
19:26 <brett@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4044.ulsfo.wmnet with OS bullseye [production]
19:25 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3052.esams.wmnet with reason: host reimage [production]
19:22 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp3052.esams.wmnet with reason: host reimage [production]
19:21 <cstone> payments-wiki upgraded from 653c7cc8 to f20a2208 [production]
19:16 <brett@cumin1001> START - Cookbook sre.hosts.reimage for host cp4044.ulsfo.wmnet with OS bullseye [production]
19:15 <brett@cumin1001> conftool action : set/pooled=yes; selector: name=cp4051.ulsfo.wmnet [production]
19:01 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp3052.esams.wmnet with OS bullseye [production]
18:46 <sukhe@cumin2002> END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['cp3052.esams.wmnet'] [production]
18:46 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp3052.esams.wmnet'] [production]
18:46 <sukhe@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp3052.esams.wmnet'] [production]
18:45 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp3052.esams.wmnet'] [production]
18:45 <sukhe@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp3052.esams.wmnet with OS bullseye [production]
18:38 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp3052.esams.wmnet with OS bullseye [production]
18:37 <sukhe@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp3052.esams.wmnet with OS bullseye [production]
18:37 <sukhe@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp3052.esams.wmnet'] [production]