4401-4450 of 10000 results (30ms)
2024-03-01 §
16:03 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on db2117.codfw.wmnet with reason: Silence for maintenance [production]
16:03 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on db2117.codfw.wmnet with reason: Silence for maintenance [production]
2024-02-29 §
22:37 <foks> removing 4 files for legal compliance [production]
21:57 <mutante> phabricator - added STran to WMF-NDA (group 61) - T355388 [production]
21:31 <mutante> phabricator - added Fring to WMF-NDA (group 61) - T358578 [production]
21:29 <mutante> phabricator - added Ifeatu_Nnaobi_WMDE to WMF-NDA (group 61) - T358578 [production]
19:35 <dduvall@deploy2002> rebuilt and synchronized wikiversions files: group2 wikis to 1.42.0-wmf.20 refs T354438 [production]
06:36 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on es1034.eqiad.wmnet with reason: Reimage [production]
06:35 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on es1034.eqiad.wmnet with reason: Reimage [production]
04:19 <tstarling@deploy2002> Synchronized wmf-config/CommonSettings.php: Switch block schema to read-old/write-both mode T355034 (duration: 08m 47s) [production]
2024-02-28 §
19:22 <dduvall@deploy2002> Synchronized php: group1 wikis to 1.42.0-wmf.20 refs T354438 (duration: 08m 37s) [production]
19:14 <dduvall@deploy2002> rebuilt and synchronized wikiversions files: group1 wikis to 1.42.0-wmf.20 refs T354438 [production]
16:31 <jayme@cumin1002> conftool action : set/pooled=yes; selector: name=mw23(2[5-9]|3[0-4]).codfw.wmnet [production]
15:35 <jayme@cumin1002> conftool action : set/pooled=inactive; selector: name=mw23(2[5-9]|3[0-4]).codfw.wmnet [production]
02:52 <swfrench-wmf> Running 'sudo systemctl start etcdmirror-conftool-eqiad-wmnet.service' on conf2005 [production]
02:50 <swfrench-wmf> Correction: Actually running 'curl https://conf2005.codfw.wmnet:2379/v2/keys/__replication/conftool -XPUT -d "value=3021126"' on conf2005 in an attempt to unwedge replication [production]
02:47 <swfrench-wmf> Running 'curl https://conf2005.codfw.wmnet:2379/v2/keys/__replication -XPUT -d "value=3021126"' on conf2005 in an attempt to unwedge replication [production]
2024-02-27 §
19:26 <dduvall@deploy2002> rebuilt and synchronized wikiversions files: group0 wikis to 1.42.0-wmf.20 refs T354438 [production]
04:56 <mwpresync@deploy2002> Finished scap: testwikis wikis to 1.42.0-wmf.20 refs T354438 (duration: 52m 18s) [production]
04:04 <mwpresync@deploy2002> Started scap: testwikis wikis to 1.42.0-wmf.20 refs T354438 [production]
04:02 <mwpresync@deploy2002> Pruned MediaWiki: 1.42.0-wmf.17 (duration: 02m 00s) [production]
2024-02-22 §
19:18 <jhuneidi@deploy2002> rebuilt and synchronized wikiversions files: group2 wikis to 1.42.0-wmf.19 refs T354437 [production]
2024-02-21 §
22:08 <dreamyjazz@deploy2002> Finished scap: Backport for [[gerrit:1005607|Pin wgGlobalBlockingAllowGlobalAccountBlocks as false on WMF wikis (T356923 T356924)]] (duration: 10m 16s) [production]
21:59 <dreamyjazz@deploy2002> dreamyjazz: Backport for [[gerrit:1005607|Pin wgGlobalBlockingAllowGlobalAccountBlocks as false on WMF wikis (T356923 T356924)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
21:58 <dreamyjazz@deploy2002> Started scap: Backport for [[gerrit:1005607|Pin wgGlobalBlockingAllowGlobalAccountBlocks as false on WMF wikis (T356923 T356924)]] [production]
20:54 <jhuneidi@deploy2002> Synchronized php: group1 wikis to 1.42.0-wmf.19 refs T354437 (duration: 08m 35s) [production]
20:46 <jhuneidi@deploy2002> rebuilt and synchronized wikiversions files: group1 wikis to 1.42.0-wmf.19 refs T354437 [production]
20:16 <jhuneidi@deploy2002> scap failed: average error rate on 4/4 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org for details) [production]
16:24 <cgoubert@cumin2002> conftool action : set/pooled=yes; selector: name=parse200(4|5).* [production]
2024-02-20 §
19:12 <jhuneidi@deploy2002> rebuilt and synchronized wikiversions files: group0 wikis to 1.42.0-wmf.19 refs T354437 [production]
16:35 <reedy@deploy2002> Synchronized php-1.42.0-wmf.19/extensions/AntiSpoof/: T357995 (duration: 11m 02s) [production]
04:56 <mwpresync@deploy2002> Finished scap: testwikis wikis to 1.42.0-wmf.19 refs T354437 (duration: 52m 09s) [production]
04:04 <mwpresync@deploy2002> Started scap: testwikis wikis to 1.42.0-wmf.19 refs T354437 [production]
04:02 <mwpresync@deploy2002> Pruned MediaWiki: 1.42.0-wmf.16 (duration: 01m 57s) [production]
02:15 <tstarling@deploy2002> Synchronized wmf-config/CommonSettings.php: Set $wgLoginNotifyUseCheckUser = false T346989 (duration: 08m 13s) [production]
2024-02-19 §
18:50 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2194:3316 (re)pooling @ 4%: Cloning to db2194 done', diff saved to https://phabricator.wikimedia.org/P57141 and previous config saved to /var/cache/conftool/dbconfig/20240219-185008-arnaudb.json [production]
16:19 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2194:3317 (re)pooling @ 4%: Cloning to db2194 done', diff saved to https://phabricator.wikimedia.org/P57111 and previous config saved to /var/cache/conftool/dbconfig/20240219-161914-arnaudb.json [production]
2024-02-16 §
15:53 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: cloudelastic100[1-4]* for decom hosts - bking@cumin2002 - T357780 [production]
15:53 <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: cloudelastic100[1-4]* for decom hosts - bking@cumin2002 - T357780 [production]
14:32 <Lucas_WMDE> STOP persistRevisionThreadItems on enwiki for T315510 – for restart on wmf.18; last output: --start '["73436010"]' [production]
2024-02-15 §
21:20 <brennen@deploy2002> rebuilt and synchronized wikiversions files: group2 wikis to 1.42.0-wmf.18 refs T354436 [production]
20:46 <brennen@deploy2002> Synchronized php: group1 wikis to 1.42.0-wmf.18 refs T354436 (duration: 08m 05s) [production]
20:38 <brennen@deploy2002> rebuilt and synchronized wikiversions files: group1 wikis to 1.42.0-wmf.18 refs T354436 [production]
19:22 <brennen@deploy2002> rebuilt and synchronized wikiversions files: group2 wikis to 1.42.0-wmf.18 refs T354436 [production]
19:04 <brennen> train 1.42.0-wmf.18 (T354436): no current blockers, rolling to all wikis. [production]
17:48 <swfrench-wmf> reenabled puppet on mediawiki::webserver hosts after deploying for T357436 [production]
17:05 <swfrench-wmf> disabling puppet shortly on mediawiki::webserver hosts to deploy T357436 [production]
2024-02-14 §
19:34 <brennen@deploy2002> Synchronized php: group1 wikis to 1.42.0-wmf.18 refs T354436 (duration: 07m 35s) [production]
19:26 <brennen@deploy2002> rebuilt and synchronized wikiversions files: group1 wikis to 1.42.0-wmf.18 refs T354436 [production]
19:14 <brennen> train 1.42.0-wmf.18 (T354436): logs chill, no current blockers, rolling to group1. [production]