|
2021-03-11
ยง
|
| 21:20 |
<brennen> |
train status: 1.36.0-wmf.34 (T274938): rolling back to group1 and marking T277229 a train blocker |
[production] |
| 21:17 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup1003.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:15 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on backup1003.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:14 |
<tgr@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:670858|Enable GrowthExperiments link recommendations on testwiki (T277173)] (duration: 00m 59s) |
[production] |
| 21:13 |
<zpapierski@deploy1002> |
Finished deploy [wikimedia/discovery/analytics@3810277]: T273847 export queries to relforge dag deployment - correct start date (duration: 01m 53s) |
[production] |
| 21:12 |
<zpapierski@deploy1002> |
Started deploy [wikimedia/discovery/analytics@3810277]: T273847 export queries to relforge dag deployment - correct start date |
[production] |
| 21:05 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mw2216.codfw.wmnet |
[production] |
| 21:04 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts mw2215.codfw.wmnet |
[production] |
| 21:03 |
<otto@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 21:03 |
<otto@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 21:03 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mw2215.codfw.wmnet |
[production] |
| 21:00 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on mw2216.codfw.wmnet with reason: decom |
[production] |
| 21:00 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on mw2216.codfw.wmnet with reason: decom |
[production] |
| 21:00 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on mw2215.codfw.wmnet with reason: decom |
[production] |
| 21:00 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on mw2215.codfw.wmnet with reason: decom |
[production] |
| 21:00 |
<otto@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 21:00 |
<otto@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 20:58 |
<mutante> |
deactivating codfw API canaries on old hardware (T277119) |
[production] |
| 20:57 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw2216.codfw.wmnet |
[production] |
| 20:57 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw2215.codfw.wmnet |
[production] |
| 20:50 |
<otto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 20:46 |
<zpapierski@deploy1002> |
Finished deploy [wikimedia/discovery/analytics@cc478d4]: T273847 export queries to relforge dag deployment (duration: 02m 09s) |
[production] |
| 20:44 |
<zpapierski@deploy1002> |
Started deploy [wikimedia/discovery/analytics@cc478d4]: T273847 export queries to relforge dag deployment |
[production] |
| 20:35 |
<otto@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventstreams-internal' for release 'main' . |
[production] |
| 20:33 |
<otto@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'eventstreams-internal' for release 'main' . |
[production] |
| 20:28 |
<otto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'eventstreams-internal' for release 'main' . |
[production] |
| 20:20 |
<mutante> |
phab1001 - systemctl start phabricator_clean_tmp_files - now Succeeded |
[production] |
| 20:17 |
<razzi@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host matomo1002.eqiad.wmnet |
[production] |
| 20:13 |
<razzi@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host matomo1002.eqiad.wmnet |
[production] |
| 20:04 |
<brennen@deploy1002> |
rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.34 |
[production] |
| 19:59 |
<mutante> |
phab1001 - sudo systemctl start phabricator_clean_tmp_files (manually run after conversion from cron to timer, and it fails with permission issues) |
[production] |
| 19:55 |
<tgr_> |
T277173 running mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=testwiki GrowthExperiments |
[production] |
| 19:54 |
<tgr@deploy1002> |
Synchronized wmf-config/: Config: [[gerrit:670857|Configure GrowthExperiments Add Link settings, step 2 (T277173)]] (duration: 01m 08s) |
[production] |
| 19:43 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 19:30 |
<tgr@deploy1002> |
Synchronized wmf-config/: Config: [[gerrit:670887|Configure GrowthExperiments Add Link settings, step 1 (T277173)]] (duration: 01m 08s) |
[production] |
| 19:18 |
<tgr@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:668196|wikitech: enable BetaFeatures (T125941)]] (duration: 01m 08s) |
[production] |
| 19:13 |
<hnowlan@deploy1002> |
Finished deploy [restbase/deploy@6f0fe23]: Remove internal ratelimits that were causing service proxy issues (duration: 16m 25s) |
[production] |
| 18:56 |
<hnowlan@deploy1002> |
Started deploy [restbase/deploy@6f0fe23]: Remove internal ratelimits that were causing service proxy issues |
[production] |
| 18:47 |
<tgr_> |
running mwscript extensions/GrowthExperiments/maintenance/importOresTopics.php testwiki --count 1000 --verbose --wikiId enwiki --apiUrl 'https://en.wikipedia.org/w/api.php' |
[production] |
| 17:31 |
<effie> |
install mecached 1.6.6-1 on mwdebug1001 |
[production] |
| 16:26 |
<effie> |
upgrade memcached on mc1021, mc2021 |
[production] |
| 16:20 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 16:14 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1110 (re)pooling @ 100%: Repool db1110 after schema change', diff saved to https://phabricator.wikimedia.org/P14802 and previous config saved to /var/cache/conftool/dbconfig/20210311-161138-root.json |
[production] |
| 15:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1110 (re)pooling @ 60%: Repool db1110 after schema change', diff saved to https://phabricator.wikimedia.org/P14801 and previous config saved to /var/cache/conftool/dbconfig/20210311-155635-root.json |
[production] |
| 15:53 |
<cmjohnson1> |
updating firmware wdqs1009 T274751 |
[production] |
| 15:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1110 (re)pooling @ 30%: Repool db1110 after schema change', diff saved to https://phabricator.wikimedia.org/P14800 and previous config saved to /var/cache/conftool/dbconfig/20210311-154131-root.json |
[production] |
| 15:35 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 15:30 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 15:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1110 (re)pooling @ 10%: Repool db1110 after schema change', diff saved to https://phabricator.wikimedia.org/P14799 and previous config saved to /var/cache/conftool/dbconfig/20210311-152627-root.json |
[production] |