2651-2700 of 10000 results (68ms)
2020-03-24 §
20:32 <twentyafterfour@deploy1001> Synchronized wmf-config: Now touch and sync again because of settings cache rache condition. refs T248409 (duration: 00m 59s) [production]
20:31 <cdanis> rebooting cr2-eqsin T248394 [production]
20:30 <twentyafterfour@deploy1001> Synchronized wmf-config: Now sync InitializeSettings* refs T248409 (duration: 00m 59s) [production]
20:28 <twentyafterfour@deploy1001> Synchronized wmf-config/CommonSettings.php: sync CommonSettings before InitialiseSettings refs T248409 (duration: 00m 58s) [production]
20:27 <volans> force rebooting analytics1044 from console, host down and unreachable (ping, ssh, console) [production]
20:26 <cdanis> commit flow-table-size on cr2-eqsin T248394 [production]
20:19 <cdanis> eqsin depooled for router maintenance at 16:15 [production]
19:29 <twentyafterfour@deploy1001> scap failed: average error rate on 4/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) [production]
19:29 <twentyafterfour> rolling back to wmf.24 due to high error rate refs T233873 [production]
19:28 <twentyafterfour@deploy1001> scap failed: average error rate on 7/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) [production]
18:49 <gehel> repooling wdqs1006, catched up on lag [production]
17:12 <hashar@deploy1001> Finished scap: testwiki to 1.35.0-wmf.25 and rebuild l10n cache # T233873 (duration: 77m 52s) [production]
17:10 <ebernhardson> update cloudelastic-chi replica counts from 2 to 1 T231517 [production]
16:41 <moritzm> installing linux-perf updates on stretch [production]
16:31 <moritzm> installing linux-perf-4.19 updates on buster [production]
15:58 <mutante> installing OS on otrs1001.eqiad.wmnet (T248028) [production]
15:54 <hashar@deploy1001> Started scap: testwiki to 1.35.0-wmf.25 and rebuild l10n cache # T233873 [production]
15:35 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
15:31 <hashar@deploy1001> Pruned MediaWiki: 1.35.0-wmf.22 (duration: 02m 02s) [production]
15:29 <hashar@deploy1001> Pruned MediaWiki: 1.35.0-wmf.21 (duration: 24m 00s) [production]
15:17 <hashar> Cleaning old MediaWiki deployments # T233873 [production]
15:03 <hashar> Applied patches to 1.35.0-wmf.25 # T233873 [production]
14:59 <hashar> scap prep 1.35.0-wmf.25 # T233873 [production]
14:55 <gehel> depooling wdqs1006 to catch up on lag [production]
14:28 <marostegui> Deploy schema change on db2117 (s6) [production]
14:26 <hashar> Branching wmf/1.35.0-wmf.25 # T233873 [production]
13:22 <moritzm> installing glib2.0 updates from Stretch point release [production]
13:04 <moritzm> installing maridb-10.1 updates from Stretch point release (client/tools/libraries as packaged by Debian, different from wmf-mariadb) [production]
12:16 <Urbanecm> mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=commonswiki --logwiki=metawiki 'Toroid~huwiki' 'Toroidt' (T248371) [production]
12:10 <Urbanecm> mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwiki --logwiki=metawiki 'Erika Greenberg' 'Copperqueen' (T248371) [production]
11:57 <Urbanecm> mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=commonswiki --logwiki=metawiki 'Romy merdeka' 'Romy_Dwi_Laksono' (T248371) [production]
11:55 <marostegui> Deploy schema change on db2087 db2089 db2097 [production]
11:34 <Urbanecm> EU SWAT done [production]
11:29 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: e28c819: Enable visualeditor on hewiktionary by default (T248311; take II) (duration: 00m 59s) [production]
11:28 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: e28c819: Enable visualeditor on hewiktionary by default (T248311) (duration: 00m 59s) [production]
11:25 <urbanecm@deploy1001> Synchronized dblists/visualeditor-nondefault.dblist: SWAT: e28c819: Enable visualeditor on hewiktionary by default (T248311) (duration: 01m 03s) [production]
10:08 <gehel> restart blazegraph and updater on wdqs1004 [production]
09:41 <marostegui> Deploy schema change on db2076 (s6) [production]
08:39 <marostegui> Rename nova database tables on db1133 (m5 master) - T248313 [production]
08:25 <marostegui> Rename wikidatawiki.wb_terms on db1104 - T248086 [production]
07:33 <elukey> restart update-openstack-mirror.service on sodium [production]
06:55 <marostegui> Reboot dbproxy1018 [production]
06:42 <marostegui> Reboot dbproxy1019 [production]
06:16 <marostegui> Create empty database testreduce on m5 master T245408 [production]
06:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Set db1087, vslow s8, with weight 1 as it originally had', diff saved to https://phabricator.wikimedia.org/P10753 and previous config saved to /var/cache/conftool/dbconfig/20200324-060133-marostegui.json [production]
2020-03-23 §
21:50 <krinkle@deploy1001> Synchronized docroot/noc/css/vector.css: I627a0ddba5 (duration: 01m 02s) [production]
21:39 <mholloway-shell@deploy1001> Finished deploy [recommendation-api/deploy@26aa5c3]: Update recommendation-api to 3141cb6 (duration: 03m 21s) [production]
18:45 <Urbanecm> Morning SWAT done [production]
18:41 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 0e535b1: InitialiseSettings - clean up groupOverrides layout / spacing (T231178; take II) (duration: 00m 59s) [production]
18:39 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 0e535b1: InitialiseSettings - clean up groupOverrides layout / spacing (T231178) (duration: 01m 00s) [production]