1451-1500 of 10000 results (37ms)
2021-10-14 ยง
23:21 <ryankemper@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814 [production]
23:11 <brennen@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:730933|Add americanantiquarian.org to the wgCopyUploadsDomains allowlist of Wikimedia Commons (T292918)]] (duration: 00m 57s) [production]
23:11 <mutante> mw1452 - re-pooled, scap pull [production]
23:09 <dzahn@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'miscweb' for release 'main' . [production]
22:35 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
22:35 <ryankemper> T288231 Ran puppet on `wdqs2006`, now back to the cookbook run [production]
22:33 <ryankemper> T288231 Forgot about running puppet-agent on `wdqs2006`; aborted cookbook run [production]
22:33 <ryankemper@cumin1001> END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) [production]
22:33 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
22:32 <ryankemper> T288231 Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/730795; proceeding to data-transfer on `wdqs2006`: `sudo rm -fv /srv/wdqs/data_loaded` on `wdqs2006` followed by `ryankemper@cumin1001:~$ sudo cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest wdqs2006.codfw.wmnet --reason "streaming updater cutover for wdqs2005" --blazegraph_instance blazegraph --task-id T288231` [production]
22:31 <mutante> depooling mw1452 for testig [production]
22:28 <ryankemper> T288231 `ryankemper@wdqs2005:~$ sudo pool`: transfer completed successfully; tests passing on host (used `ssh -L 9999:localhost:80 wdqs2005.codfw.wmnet` to establish tunnel) [production]
22:23 <dpifke@deploy1002> Finished deploy [performance/arc-lamp@84fe496]: New flamegraph.pl from upstream T291898 (duration: 00m 05s) [production]
22:23 <dpifke@deploy1002> Started deploy [performance/arc-lamp@84fe496]: New flamegraph.pl from upstream T291898 [production]
22:17 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814 [production]
22:07 <eileen> civicrm revision changed from 018d3b19fe to 9b5e0d015b, config revision is 781d6a1b1f [production]
21:34 <robh@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:25 <robh@cumin1001> START - Cookbook sre.dns.netbox [production]
21:10 <robh@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
21:06 <robh@cumin1001> START - Cookbook sre.dns.netbox [production]
19:45 <dancy@deploy1002> rebuilt and synchronized wikiversions files: group2 wikis to 1.38.0-wmf.4 refs T281168 [production]
19:23 <dzahn@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'miscweb' for release 'main' . [production]
19:05 <dzahn@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'miscweb' for release 'main' . [production]
18:53 <dzahn@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'miscweb' for release 'main' . [production]
18:53 <urbanecm> [urbanecm@mwmaint1002 ~]$ mwscript namespaceDupes.php --wiki=dagwiki --fix [production]
18:47 <urbanecm> [urbanecm@mwmaint1002 ~]$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=frwiktionary --logwiki=metawiki 'TURK FASTER' 'ARTHUR MORGAN' [production]
18:42 <urbanecm> [urbanecm@mwmaint1002 ~]$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=metawiki --logwiki=metawiki 'George Dum Fulton' 'George Fulton' # T293403 [production]
18:41 <urbanecm> UTC evening B&C done [production]
18:40 <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.4/extensions/MediaSearch/extension.json: 6da3523daaba85a4199721980c0a9c96b20697e7: Fix assessment quickview labels (T292596) (duration: 01m 03s) [production]
18:37 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: c8dffefd0d095abe3709dcc962d5d24f27b55869: Create Salima namespace for dagwiki (T289911) (duration: 01m 04s) [production]
18:30 <dzahn@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'miscweb' for release 'main' . [production]
18:25 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 0bccd4bc45498db8628567574d0bb3a23f8fb378: Add $wgSitename and $wgMetaNamespace for kswiki and kswiktionary (T289752, T289767) (duration: 01m 04s) [production]
18:17 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
18:14 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 262e588b44f126fb9e1aa933a3ca59b191b42bd7: Enable Growth mentor dashboard backend on all wikis (T278920) (duration: 01m 05s) [production]
18:07 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 41baa8c41d64510986f009b9be2d70dad0915f8c: Add new mediawiki.skin_diff event logging stream (T289622) (duration: 01m 05s) [production]
18:03 <addshore@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
18:02 <addshore@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
18:01 <addshore@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . [production]
17:54 <bd808@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'toolhub' for release 'main' . [production]
17:52 <rzl> repooled mw1452 (with `sudo pool` so no auto log from conftool) [production]
17:47 <bd808@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'toolhub' for release 'main' . [production]
17:45 <rzl@cumin1001> conftool action : set/pooled=no; selector: name=mw1452.eqiad.wmnet [production]
17:42 <rzl> depool mw1452 for training [production]
17:32 <addshore@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
17:31 <addshore@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
17:29 <addshore@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . [production]
16:44 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
16:44 <ryankemper> T288231 Manually killed dangling `pigz` / `nc` processes on `wdqs2008` (and `wdqs2005` implicitly). Should be in the right state to re-start the `data-transfer` cookbook from again [production]
16:41 <ryankemper@cumin1001> END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) [production]
16:37 <elukey> drop kubeflow-kfserving* docker images from deneb [production]