|
2020-07-07
ยง
|
| 15:27 |
<elukey> |
root-tmux on cumin1001 - cumin 'c:profile::mediawiki::mcrouter_wancache' '/usr/local/sbin/restart-mcrouter' -b 2 -s 5 - roll restart of mw-mcrouter to pick up new settings - T255511 |
[production] |
| 15:13 |
<hnowlan@deploy1001> |
Started restart [restbase/deploy@05b8bd5]: Restarting restbase after removal of restbase2009 |
[production] |
| 15:12 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' . |
[production] |
| 15:12 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-main' for release 'production' . |
[production] |
| 15:09 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . |
[production] |
| 15:09 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . |
[production] |
| 15:06 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' . |
[production] |
| 15:04 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 15:04 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 15:02 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-main' for release 'production' . |
[production] |
| 15:02 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' . |
[production] |
| 15:01 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' . |
[production] |
| 14:58 |
<hashar@deploy1001> |
Finished deploy [integration/docroot@708d3eb]: Second deployment to ensure everything works fine. Thank you jynus (duration: 00m 04s) |
[production] |
| 14:58 |
<hashar@deploy1001> |
Started deploy [integration/docroot@708d3eb]: Second deployment to ensure everything works fine. Thank you jynus |
[production] |
| 14:53 |
<_joe_> |
restarted restbase on restbase2022 after removing restbase2009 from the cassandra seeds |
[production] |
| 14:48 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . |
[production] |
| 14:47 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . |
[production] |
| 14:38 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 14:38 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 14:31 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-main' for release 'production' . |
[production] |
| 14:31 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' . |
[production] |
| 14:30 |
<papaul> |
replacing msw-a5,a6,a7 and a8 |
[production] |
| 14:30 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' . |
[production] |
| 14:24 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . |
[production] |
| 14:24 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . |
[production] |
| 14:20 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 14:20 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 14:16 |
<hashar@deploy1001> |
Finished deploy [integration/docroot@708d3eb]: (no justification provided) (duration: 00m 09s) |
[production] |
| 14:16 |
<hashar@deploy1001> |
Started deploy [integration/docroot@708d3eb]: (no justification provided) |
[production] |
| 13:38 |
<_joe_> |
rolling restart of restbase to pick up using envoy |
[production] |
| 13:31 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 13:31 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 13:29 |
<XioNoX> |
cr2-eqiad> request vmhost snapshot routing-engine both - T257153 |
[production] |
| 13:24 |
<XioNoX> |
cr1-eqiad> request vmhost snapshot routing-engine both - T257153 |
[production] |
| 13:15 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Promote es2021 to es4 master T257284', diff saved to https://phabricator.wikimedia.org/P11789 and previous config saved to /var/cache/conftool/dbconfig/20200707-131524-kormat.json |
[production] |
| 12:59 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:57 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:44 |
<kormat> |
starting (codfw) es5 failover from es2020 to es2021 T257284 |
[production] |
| 12:30 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Set es2021 to weight 50 T257284', diff saved to https://phabricator.wikimedia.org/P11787 and previous config saved to /var/cache/conftool/dbconfig/20200707-123003-kormat.json |
[production] |
| 12:12 |
<jforrester@deploy1001> |
Finished scap: Full scap and testwikis to 1.35.0-wmf.40 for T256668 (duration: 33m 09s) |
[production] |
| 12:01 |
<marostegui> |
Deploy schema change on labswiki (wikitech) master - T253276 |
[production] |
| 11:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1082', diff saved to https://phabricator.wikimedia.org/P11786 and previous config saved to /var/cache/conftool/dbconfig/20200707-115838-marostegui.json |
[production] |
| 11:39 |
<jforrester@deploy1001> |
Started scap: Full scap and testwikis to 1.35.0-wmf.40 for T256668 |
[production] |
| 11:38 |
<jforrester@deploy1001> |
scap failed: LockFailedError Failed to acquire lock "/var/lock/scap.operations_mediawiki-config.lock"; owner is "jforrester"; reason is "testwikis wikis to 1.35.0-wmf.40" (duration: 00m 00s) |
[production] |
| 11:33 |
<moritzm> |
installing PHP 7.0 security updates |
[production] |
| 11:29 |
<marostegui> |
Deploy schema change on db1082, this will create lag on s5 labs |
[production] |
| 11:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1082', diff saved to https://phabricator.wikimedia.org/P11784 and previous config saved to /var/cache/conftool/dbconfig/20200707-112926-marostegui.json |
[production] |
| 11:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1130', diff saved to https://phabricator.wikimedia.org/P11783 and previous config saved to /var/cache/conftool/dbconfig/20200707-112830-marostegui.json |
[production] |
| 11:26 |
<godog> |
test bumping logstash7 batch size to 256 |
[production] |
| 11:17 |
<moritzm> |
prune PHP 7.0 packages from mwdebug1001/2001/2002 |
[production] |