3351-3400 of 10000 results (80ms)
2023-01-31 §
08:45 <elukey> restore previously removed password for keystore to kafka-logging clusters [production]
08:39 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . [production]
08:36 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
07:56 <moritzm> installing bash bugfix updates from Bullseye point release [production]
07:22 <marostegui> dbmaint Schema change on s3 eqiad T328373 [production]
07:22 <marostegui> dbmaint Schema change on s1 eqiad T328373 [production]
07:10 <marostegui> Failover m2 from db1164 to db1195 - T328253 [production]
07:06 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2133,2160].codfw.wmnet,db[1117,1164,1195].eqiad.wmnet with reason: Primary switchover m2 T328253 [production]
07:06 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on db[2133,2160].codfw.wmnet,db[1117,1164,1195].eqiad.wmnet with reason: Primary switchover m2 T328253 [production]
07:03 <marostegui> dbmaint Schema change on s5 eqiad T328373 [production]
06:59 <marostegui> dbmaint Schema change on s7 eqiad T328373 [production]
06:57 <marostegui> dbmaint Schema change on s4 eqiad T328373 [production]
06:52 <marostegui> dbmaint Schema change on s8 eqiad T328373 [production]
05:02 <mwpresync@deploy1002> Pruned MediaWiki: 1.40.0-wmf.19 (duration: 02m 15s) [production]
05:02 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
05:00 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
04:55 <mwpresync@deploy1002> Finished scap: testwikis wikis to 1.40.0-wmf.21 refs T325584 (duration: 52m 56s) [production]
04:02 <mwpresync@deploy1002> Started scap: testwikis wikis to 1.40.0-wmf.21 refs T325584 [production]
02:44 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp3053.esams.wmnet,service=ats-be [production]
02:43 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp3053.esams.wmnet,service=cdn [production]
02:28 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3053.esams.wmnet with OS bullseye [production]
02:02 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3053.esams.wmnet with reason: host reimage [production]
01:59 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp3053.esams.wmnet with reason: host reimage [production]
01:37 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp3053.esams.wmnet with OS bullseye [production]
01:33 <sukhe@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp3053.esams.wmnet'] [production]
01:31 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp3053.esams.wmnet'] [production]
00:50 <brett@cumin1001> conftool action : set/pooled=yes; selector: name=cp5027.eqsin.wmnet [production]
00:42 <brett@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5027.eqsin.wmnet with OS bullseye [production]
00:14 <mutante> etherpad - maintenance downtime for about 5 minutes to test monitoring [production]
00:09 <brett@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5027.eqsin.wmnet with reason: host reimage [production]
00:06 <brett@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp5027.eqsin.wmnet with reason: host reimage [production]
2023-01-30 §
23:30 <brett@cumin1001> START - Cookbook sre.hosts.reimage for host cp5027.eqsin.wmnet with OS bullseye [production]
23:29 <brett@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp3053.esams.wmnet with OS bullseye [production]
23:07 <brett@cumin1001> START - Cookbook sre.hosts.reimage for host cp3053.esams.wmnet with OS bullseye [production]
22:58 <brett@cumin1001> conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet [production]
22:50 <brett@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp3053.esams.wmnet with OS bullseye [production]
22:38 <brett@cumin1001> START - Cookbook sre.hosts.reimage for host cp3053.esams.wmnet with OS bullseye [production]
22:36 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp2030.codfw.wmnet,service=ats-be [production]
22:36 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp2030.codfw.wmnet,service=cdn [production]
22:11 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2030.codfw.wmnet with OS bullseye [production]
21:56 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:884138|Try to determine what's adding to Parsoid init times (T328201)]], [[gerrit:885008|Update interwiki cache]] (duration: 12m 24s) [production]
21:47 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2030.codfw.wmnet with reason: host reimage [production]
21:46 <urbanecm@deploy1002> arlolra and urbanecm: Backport for [[gerrit:884138|Try to determine what's adding to Parsoid init times (T328201)]], [[gerrit:885008|Update interwiki cache]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet [production]
21:44 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:884138|Try to determine what's adding to Parsoid init times (T328201)]], [[gerrit:885008|Update interwiki cache]] [production]
21:43 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp2030.codfw.wmnet with reason: host reimage [production]
21:42 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:884153|GrowthExperiments: Update campaign configuration (T321370)]] (duration: 08m 47s) [production]
21:35 <urbanecm@deploy1002> tgr and urbanecm: Backport for [[gerrit:884153|GrowthExperiments: Update campaign configuration (T321370)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
21:34 <eevans@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase2020.codfw.wmnet: Replace Cassandra keys & certs - eevans@cumin1001 [production]
21:34 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:884153|GrowthExperiments: Update campaign configuration (T321370)]] [production]
21:33 <brett@cumin1001> conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet [production]