651-700 of 10000 results (35ms)
2021-03-31 §
02:17 <pt1979@cumin2001> START - Cookbook sre.dns.netbox [production]
02:06 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on wcqs2003.codfw.wmnet with reason: REIMAGE [production]
02:05 <pt1979@cumin2001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
02:04 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs2003.codfw.wmnet with reason: REIMAGE [production]
02:00 <pt1979@cumin2001> START - Cookbook sre.dns.netbox [production]
01:37 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on wcqs2002.codfw.wmnet with reason: REIMAGE [production]
01:35 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs2002.codfw.wmnet with reason: REIMAGE [production]
01:15 <urbanecm@deploy1002> Synchronized wmf-config/config/gawiki.yaml: 3283ae59f25f02966a81ed2f0b51b964f733cf65: Enable local uploads on Irish Wikipedia (T277723) (duration: 01m 08s) [production]
01:13 <urbanecm@deploy1002> Synchronized dblists/commonsuploads.dblist: 3283ae59f25f02966a81ed2f0b51b964f733cf65: Enable local uploads on Irish Wikipedia (T277723) (duration: 01m 08s) [production]
01:07 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on wcqs2001.codfw.wmnet with reason: REIMAGE [production]
01:05 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs2001.codfw.wmnet with reason: REIMAGE [production]
2021-03-30 §
23:59 <Trey314159> reindexing English wikis on elastic@eqiad, elastic@codfw, and cloudelastic (T274200) [production]
23:55 <legoktm@deploy1002> Synchronized php-1.36.0-wmf.37/extensions/TimedMediaHandler/extension.json: Allow autoconfirmed users to see Special:TranscodeStatistics by default (T278867) (duration: 01m 08s) [production]
23:53 <legoktm@deploy1002> Synchronized php-1.36.0-wmf.36/extensions/TimedMediaHandler/extension.json: Allow autoconfirmed users to see Special:TranscodeStatistics by default (T278867) (duration: 01m 08s) [production]
23:29 <Amir1> sudo django-admin hyperkitty_import -l discovery-alerts@lists-next.wikimedia.org discovery-alerts.mbox/discovery-alerts.mbox --pythonpath /usr/share/mailman3-web --settings settings (T278609) [production]
23:27 <pt1979@cumin2001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
23:23 <pt1979@cumin2001> START - Cookbook sre.dns.netbox [production]
23:15 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: ef306a35464f295f43b874301cf0170edcfa4d8c: Growth features: bnwiki: Enable impact module (T274793) (duration: 01m 07s) [production]
22:52 <cstone> civicrm revision changed from ad430721f6 to 7040b68c11 [production]
21:11 <twentyafterfour@deploy1002> Finished deploy [releng/phatality@fbca60c]: rollback (duration: 00m 12s) [production]
21:11 <twentyafterfour@deploy1002> Started deploy [releng/phatality@fbca60c]: rollback [production]
21:05 <twentyafterfour@deploy1002> Finished deploy [releng/phatality@fbca60c]: trying again with newly built zip (duration: 00m 12s) [production]
21:05 <twentyafterfour@deploy1002> Started deploy [releng/phatality@fbca60c]: trying again with newly built zip [production]
21:02 <legoktm> scap pulling on mw1298 [production]
20:59 <twentyafterfour@deploy1002> Finished deploy [releng/phatality@715d809]: (no justification provided) (duration: 00m 15s) [production]
20:58 <twentyafterfour@deploy1002> Started deploy [releng/phatality@715d809]: (no justification provided) [production]
20:58 <legoktm> killed remaining ffmpeg on mw1298 [production]
20:56 <twentyafterfour@deploy1002> Finished deploy [releng/phatality@715d809]: (no justification provided) (duration: 00m 12s) [production]
20:56 <twentyafterfour@deploy1002> Started deploy [releng/phatality@715d809]: (no justification provided) [production]
20:53 <kharlan@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
20:52 <kharlan@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
20:41 <twentyafterfour@deploy1002> Finished deploy [releng/phatality@715d809]: (no justification provided) (duration: 00m 20s) [production]
20:41 <twentyafterfour@deploy1002> Started deploy [releng/phatality@715d809]: (no justification provided) [production]
20:41 <kharlan@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . [production]
20:40 <kharlan@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
20:38 <kharlan@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
20:37 <kharlan@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . [production]
20:37 <twentyafterfour@deploy1002> Finished deploy [releng/phatality@715d809]: (no justification provided) (duration: 00m 31s) [production]
20:36 <twentyafterfour@deploy1002> Started deploy [releng/phatality@715d809]: (no justification provided) [production]
20:35 <twentyafterfour@deploy1002> Finished deploy [releng/phatality@715d809]: (no justification provided) (duration: 00m 05s) [production]
20:35 <twentyafterfour@deploy1002> Started deploy [releng/phatality@715d809]: (no justification provided) [production]
20:34 <kharlan@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . [production]
20:34 <twentyafterfour@deploy1002> Started restart [releng/phatality@715d809]: (no justification provided) [production]
20:33 <twentyafterfour@deploy1002> Finished scap: testwikis wikis to 1.36.0-wmf.37 refs T278343 (duration: 80m 32s) [production]
20:29 <twentyafterfour@deploy1002> Finished deploy [releng/phatality@715d809]: (no justification provided) (duration: 00m 49s) [production]
20:29 <twentyafterfour@deploy1002> Started deploy [releng/phatality@715d809]: (no justification provided) [production]
20:28 <legoktm@deploy1002> conftool action : set/pooled=yes; selector: cluster=videoscaler,name=mw1307.eqiad.wmnet [production]
20:28 <legoktm@deploy1002> conftool action : set/pooled=yes; selector: cluster=videoscaler,name=mw1306.eqiad.wmnet [production]
20:28 <legoktm@deploy1002> conftool action : set/pooled=yes; selector: cluster=videoscaler,name=mw1305.eqiad.wmnet [production]
20:28 <legoktm@deploy1002> conftool action : set/pooled=yes; selector: cluster=videoscaler,name=mw1304.eqiad.wmnet [production]