|
2021-03-11
ยง
|
| 22:48 |
<dzahn@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 22:47 |
<mutante> |
running DNS cookbook in an attempt to remove mw2216 |
[production] |
| 22:47 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts mw2216.codfw.wmnet |
[production] |
| 22:41 |
<brennen@deploy1002> |
rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.34 |
[production] |
| 22:36 |
<brennen> |
train status: 1.36.0-wmf.34 (T274938): T277229 and T266517 related issues hopefully resolved, rolling forward to all wikis |
[production] |
| 22:34 |
<brennen@deploy1002> |
Synchronized php-1.36.0-wmf.34/extensions/WikimediaEvents/modules/ext.wikimediaEvents/clientError.js: Backport: [[gerrit:670879|Do not log script errors without file uri (T266517)]] (duration: 01m 07s) |
[production] |
| 22:33 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 22:30 |
<brennen@deploy1002> |
Synchronized php-1.36.0-wmf.34/extensions/MobileFrontend/includes/: Backport: [[gerrit:670877|Revert "Fix: Save user options only once when Advanced Mode is toggled" (T277229)]] (duration: 01m 09s) |
[production] |
| 22:28 |
<dzahn@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 21:57 |
<Amir1> |
run populate pages in cognate (T259360) |
[production] |
| 21:28 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2222.codfw.wmnet |
[production] |
| 21:27 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2223.codfw.wmnet |
[production] |
| 21:27 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2221.codfw.wmnet |
[production] |
| 21:27 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2220.codfw.wmnet |
[production] |
| 21:21 |
<brennen@deploy1002> |
rebuilt and synchronized wikiversions files: Revert "all wikis to 1.36.0-wmf.34" |
[production] |
| 21:20 |
<brennen> |
train status: 1.36.0-wmf.34 (T274938): rolling back to group1 and marking T277229 a train blocker |
[production] |
| 21:17 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup1003.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:15 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on backup1003.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:14 |
<tgr@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:670858|Enable GrowthExperiments link recommendations on testwiki (T277173)] (duration: 00m 59s) |
[production] |
| 21:13 |
<zpapierski@deploy1002> |
Finished deploy [wikimedia/discovery/analytics@3810277]: T273847 export queries to relforge dag deployment - correct start date (duration: 01m 53s) |
[production] |
| 21:12 |
<zpapierski@deploy1002> |
Started deploy [wikimedia/discovery/analytics@3810277]: T273847 export queries to relforge dag deployment - correct start date |
[production] |
| 21:05 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mw2216.codfw.wmnet |
[production] |
| 21:04 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts mw2215.codfw.wmnet |
[production] |
| 21:03 |
<otto@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 21:03 |
<otto@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 21:03 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mw2215.codfw.wmnet |
[production] |
| 21:00 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on mw2216.codfw.wmnet with reason: decom |
[production] |
| 21:00 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on mw2216.codfw.wmnet with reason: decom |
[production] |
| 21:00 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on mw2215.codfw.wmnet with reason: decom |
[production] |
| 21:00 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on mw2215.codfw.wmnet with reason: decom |
[production] |
| 21:00 |
<otto@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 21:00 |
<otto@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 20:58 |
<mutante> |
deactivating codfw API canaries on old hardware (T277119) |
[production] |
| 20:57 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw2216.codfw.wmnet |
[production] |
| 20:57 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw2215.codfw.wmnet |
[production] |
| 20:50 |
<otto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 20:46 |
<zpapierski@deploy1002> |
Finished deploy [wikimedia/discovery/analytics@cc478d4]: T273847 export queries to relforge dag deployment (duration: 02m 09s) |
[production] |
| 20:44 |
<zpapierski@deploy1002> |
Started deploy [wikimedia/discovery/analytics@cc478d4]: T273847 export queries to relforge dag deployment |
[production] |
| 20:35 |
<otto@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventstreams-internal' for release 'main' . |
[production] |
| 20:33 |
<otto@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'eventstreams-internal' for release 'main' . |
[production] |
| 20:28 |
<otto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'eventstreams-internal' for release 'main' . |
[production] |
| 20:20 |
<mutante> |
phab1001 - systemctl start phabricator_clean_tmp_files - now Succeeded |
[production] |
| 20:17 |
<razzi@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host matomo1002.eqiad.wmnet |
[production] |
| 20:13 |
<razzi@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host matomo1002.eqiad.wmnet |
[production] |
| 20:04 |
<brennen@deploy1002> |
rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.34 |
[production] |
| 19:59 |
<mutante> |
phab1001 - sudo systemctl start phabricator_clean_tmp_files (manually run after conversion from cron to timer, and it fails with permission issues) |
[production] |
| 19:55 |
<tgr_> |
T277173 running mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=testwiki GrowthExperiments |
[production] |
| 19:54 |
<tgr@deploy1002> |
Synchronized wmf-config/: Config: [[gerrit:670857|Configure GrowthExperiments Add Link settings, step 2 (T277173)]] (duration: 01m 08s) |
[production] |
| 19:43 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 19:30 |
<tgr@deploy1002> |
Synchronized wmf-config/: Config: [[gerrit:670887|Configure GrowthExperiments Add Link settings, step 1 (T277173)]] (duration: 01m 08s) |
[production] |