2451-2500 of 10000 results (106ms)
2024-06-14 §
03:39 <cdobbins@cumin1002> END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_eqsin [production]
03:39 <cdobbins@cumin1002> END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_eqsin [production]
01:07 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1229 (T352010)', diff saved to https://phabricator.wikimedia.org/P64896 and previous config saved to /var/cache/conftool/dbconfig/20240614-010717-ladsgroup.json [production]
01:07 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance [production]
01:06 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance [production]
2024-06-13 §
23:56 <zabe@deploy1002> Finished scap: T361041, [[gerrit:1043311|Update interwiki cache]] (duration: 11m 07s) [production]
23:48 <foks> removing 7 files for legal compliance [production]
23:45 <zabe@deploy1002> Started scap: T361041, [[gerrit:1043311|Update interwiki cache]] [production]
23:23 <zabe> zabe@mwmaint1002:~$ mwscript extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=sysop_plwiki --cluster=all 2>&1 | tee /tmp/sysop_plwiki.UpdateSearchIndexConfig.log # T361041 [production]
23:20 <zabe@deploy1002> Finished scap: T361041 (duration: 11m 36s) [production]
23:17 <foks> removing 9 files for legal compliance [production]
23:08 <zabe@deploy1002> Started scap: T361041 [production]
23:06 <zabe@deploy1002> Sync cancelled. [production]
23:02 <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:aqs-eqiad: Upgrade to Java 11 — T350567 - eevans@cumin1002 [production]
23:01 <zabe@deploy1002> zabe: T361041 synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
22:59 <zabe@deploy1002> Started scap: T361041 [production]
22:49 <zabe> create plwiki sysop wiki # T361041 [production]
22:37 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-ctrl2003.mgmt.codfw.wmnet with reboot policy FORCED [production]
22:05 <ryankemper@cumin2002> START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster [production]
21:33 <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-eqiad: Upgrade to Java 11 — T350567 - eevans@cumin1002 [production]
21:32 <jsn@deploy1002> Finished scap: Backport for [[gerrit:1041699|Deploy QuickSurvey for Automoderator patroller workstream survey (T362969)]] (duration: 14m 18s) [production]
21:23 <jsn@deploy1002> jsn, kgraessle: Continuing with sync [production]
21:22 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2220 (T364069)', diff saved to https://phabricator.wikimedia.org/P64894 and previous config saved to /var/cache/conftool/dbconfig/20240613-212230-marostegui.json [production]
21:20 <jsn@deploy1002> jsn, kgraessle: Backport for [[gerrit:1041699|Deploy QuickSurvey for Automoderator patroller workstream survey (T362969)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
21:17 <jsn@deploy1002> Started scap: Backport for [[gerrit:1041699|Deploy QuickSurvey for Automoderator patroller workstream survey (T362969)]] [production]
21:15 <jsn@deploy1002> Finished scap: Backport for [[gerrit:1043110|Look for iPadOS in user-agent, in addition to iOS. (T362723)]] (duration: 14m 11s) [production]
21:07 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P64893 and previous config saved to /var/cache/conftool/dbconfig/20240613-210723-marostegui.json [production]
21:07 <jsn@deploy1002> dbrant, jsn: Continuing with sync [production]
21:04 <jsn@deploy1002> dbrant, jsn: Backport for [[gerrit:1043110|Look for iPadOS in user-agent, in addition to iOS. (T362723)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
21:04 <topranks> changing BGP aggregate contribution policy / external route announcement cr2-eqdfw (T367439) [production]
21:03 <topranks> changing BGP aggregate contribution policy / external route announcement cr2-eqord (T367439) [production]
21:01 <jsn@deploy1002> Started scap: Backport for [[gerrit:1043110|Look for iPadOS in user-agent, in addition to iOS. (T362723)]] [production]
20:55 <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:aqs-codfw: Upgrade to Java 11 — T350567 - eevans@cumin1002 [production]
20:52 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P64892 and previous config saved to /var/cache/conftool/dbconfig/20240613-205215-marostegui.json [production]
20:50 <cdobbins@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp4038.eqsin.wmnet [production]
20:44 <cdobbins@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4038.ulsfo.wmnet with OS bullseye [production]
20:37 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2220 (T364069)', diff saved to https://phabricator.wikimedia.org/P64891 and previous config saved to /var/cache/conftool/dbconfig/20240613-203708-marostegui.json [production]
20:17 <cdobbins@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4038.ulsfo.wmnet with reason: host reimage [production]
20:14 <cdobbins@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp4038.ulsfo.wmnet with reason: host reimage [production]
20:13 <foks> removing 1 file for legal compliance [production]
20:00 <kamila@cumin1002> conftool action : set/pooled=yes; selector: name=wikikube-ctrl1003.eqiad.wmnet [production]
19:59 <foks> removing 2 files for legal compliance [production]
19:58 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-ctrl1003.eqiad.wmnet [production]
19:58 <kamila@cumin1002> START - Cookbook sre.hosts.remove-downtime for wikikube-ctrl1003.eqiad.wmnet [production]
19:53 <cdobbins@cumin2002> START - Cookbook sre.hosts.reimage for host cp4038.ulsfo.wmnet with OS bullseye [production]
19:51 <foks> removing 2 files for legal compliance [production]
19:51 <cdobbins@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4038.ulsfo.wmnet with OS bullseye [production]
19:41 <foks> removing 2 files for legal compliance [production]
19:28 <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-codfw: Upgrade to Java 11 — T350567 - eevans@cumin1002 [production]
19:27 <eevans@cumin1002> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for aqs1013.eqiad.wmnet [production]