6601-6650 of 10000 results (92ms)
2023-01-31 §
02:02 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3053.esams.wmnet with reason: host reimage [production]
01:59 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp3053.esams.wmnet with reason: host reimage [production]
01:37 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp3053.esams.wmnet with OS bullseye [production]
01:33 <sukhe@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp3053.esams.wmnet'] [production]
01:31 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp3053.esams.wmnet'] [production]
00:50 <brett@cumin1001> conftool action : set/pooled=yes; selector: name=cp5027.eqsin.wmnet [production]
00:42 <brett@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5027.eqsin.wmnet with OS bullseye [production]
00:14 <mutante> etherpad - maintenance downtime for about 5 minutes to test monitoring [production]
00:09 <brett@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5027.eqsin.wmnet with reason: host reimage [production]
00:06 <brett@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp5027.eqsin.wmnet with reason: host reimage [production]
2023-01-30 §
23:30 <brett@cumin1001> START - Cookbook sre.hosts.reimage for host cp5027.eqsin.wmnet with OS bullseye [production]
23:29 <brett@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp3053.esams.wmnet with OS bullseye [production]
23:07 <brett@cumin1001> START - Cookbook sre.hosts.reimage for host cp3053.esams.wmnet with OS bullseye [production]
22:58 <brett@cumin1001> conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet [production]
22:50 <brett@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp3053.esams.wmnet with OS bullseye [production]
22:38 <brett@cumin1001> START - Cookbook sre.hosts.reimage for host cp3053.esams.wmnet with OS bullseye [production]
22:36 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp2030.codfw.wmnet,service=ats-be [production]
22:36 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp2030.codfw.wmnet,service=cdn [production]
22:11 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2030.codfw.wmnet with OS bullseye [production]
21:56 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:884138|Try to determine what's adding to Parsoid init times (T328201)]], [[gerrit:885008|Update interwiki cache]] (duration: 12m 24s) [production]
21:47 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2030.codfw.wmnet with reason: host reimage [production]
21:46 <urbanecm@deploy1002> arlolra and urbanecm: Backport for [[gerrit:884138|Try to determine what's adding to Parsoid init times (T328201)]], [[gerrit:885008|Update interwiki cache]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet [production]
21:44 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:884138|Try to determine what's adding to Parsoid init times (T328201)]], [[gerrit:885008|Update interwiki cache]] [production]
21:43 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp2030.codfw.wmnet with reason: host reimage [production]
21:42 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:884153|GrowthExperiments: Update campaign configuration (T321370)]] (duration: 08m 47s) [production]
21:35 <urbanecm@deploy1002> tgr and urbanecm: Backport for [[gerrit:884153|GrowthExperiments: Update campaign configuration (T321370)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
21:34 <eevans@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase2020.codfw.wmnet: Replace Cassandra keys & certs - eevans@cumin1001 [production]
21:34 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:884153|GrowthExperiments: Update campaign configuration (T321370)]] [production]
21:33 <brett@cumin1001> conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet [production]
21:31 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:883301|Enable WelcomeSurvey at viwiki (T325376)]], [[gerrit:884930|Fix grid blowout with limited width turned off (T327423)]], [[gerrit:884929|Support new style of table of contents (T327942)]] (duration: 09m 52s) [production]
21:26 <brett@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS bullseye [production]
21:25 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp2030.codfw.wmnet with OS bullseye [production]
21:24 <eevans@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching restbase2020.codfw.wmnet: Replace Cassandra keys & certs - eevans@cumin1001 [production]
21:23 <urbanecm@deploy1002> tgr and urbanecm and jdlrobson and legoktm: Backport for [[gerrit:883301|Enable WelcomeSurvey at viwiki (T325376)]], [[gerrit:884930|Fix grid blowout with limited width turned off (T327423)]], [[gerrit:884929|Support new style of table of contents (T327942)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
21:21 <eevans@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase2019.codfw.wmnet: Replace Cassandra keys & certs - eevans@cumin1001 [production]
21:21 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:883301|Enable WelcomeSurvey at viwiki (T325376)]], [[gerrit:884930|Fix grid blowout with limited width turned off (T327423)]], [[gerrit:884929|Support new style of table of contents (T327942)]] [production]
21:21 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:884474|InitialiseSettings: add zhwiki to wgPageAssessmentsSubprojects (T326387)]] (duration: 19m 51s) [production]
21:11 <eevans@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching restbase2019.codfw.wmnet: Replace Cassandra keys & certs - eevans@cumin1001 [production]
21:03 <urbanecm@deploy1002> urbanecm and musikanimal: Backport for [[gerrit:884474|InitialiseSettings: add zhwiki to wgPageAssessmentsSubprojects (T326387)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
21:02 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp2033.codfw.wmnet,service=ats-be [production]
21:02 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp2033.codfw.wmnet,service=cdn [production]
21:01 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:884474|InitialiseSettings: add zhwiki to wgPageAssessmentsSubprojects (T326387)]] [production]
20:59 <brett@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage [production]
20:56 <brett@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage [production]
20:51 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
20:50 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
20:35 <brett@cumin1001> START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye [production]
20:35 <brett@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4052.ulsfo.wmnet with OS bullseye [production]
20:26 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2033.codfw.wmnet with OS bullseye [production]
20:23 <zabe@deploy1002> Finished scap: Backport for [[gerrit:885002|slwiki: Raise AF emergency disable treshold+count (T328366)]] (duration: 07m 32s) [production]