|
2020-07-29
§
|
| 08:05 |
<marostegui> |
Deploy MCR schema change on db1121 (lag will show up on s4), also remove triggers on db1124:3314 |
[production] |
| 08:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1121', diff saved to https://phabricator.wikimedia.org/P12111 and previous config saved to /var/cache/conftool/dbconfig/20200729-080442-marostegui.json |
[production] |
| 08:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1141', diff saved to https://phabricator.wikimedia.org/P12110 and previous config saved to /var/cache/conftool/dbconfig/20200729-080318-marostegui.json |
[production] |
| 07:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1141', diff saved to https://phabricator.wikimedia.org/P12109 and previous config saved to /var/cache/conftool/dbconfig/20200729-075558-marostegui.json |
[production] |
| 07:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1141', diff saved to https://phabricator.wikimedia.org/P12108 and previous config saved to /var/cache/conftool/dbconfig/20200729-074828-marostegui.json |
[production] |
| 07:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1141', diff saved to https://phabricator.wikimedia.org/P12107 and previous config saved to /var/cache/conftool/dbconfig/20200729-074414-marostegui.json |
[production] |
| 06:50 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:48 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:26 |
<XioNoX> |
standardize mr1-eqiad interfaces |
[production] |
| 06:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1112', diff saved to https://phabricator.wikimedia.org/P12106 and previous config saved to /var/cache/conftool/dbconfig/20200729-062224-marostegui.json |
[production] |
| 06:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1078', diff saved to https://phabricator.wikimedia.org/P12105 and previous config saved to /var/cache/conftool/dbconfig/20200729-062009-marostegui.json |
[production] |
| 06:16 |
<XioNoX> |
standardize mr1-codfw interfaces |
[production] |
| 06:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1078', diff saved to https://phabricator.wikimedia.org/P12104 and previous config saved to /var/cache/conftool/dbconfig/20200729-061450-marostegui.json |
[production] |
| 06:05 |
<XioNoX> |
standardize mr1-ulsfo interfaces |
[production] |
| 06:01 |
<legoktm> |
ssh doc1001.eqiad.wmnet sudo -u doc-uploader git -C /srv/docroot pull |
[production] |
| 05:52 |
<XioNoX> |
standardize mr1-eqsin interfaces |
[production] |
| 05:45 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 05:43 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 05:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1078', diff saved to https://phabricator.wikimedia.org/P12103 and previous config saved to /var/cache/conftool/dbconfig/20200729-050346-marostegui.json |
[production] |
| 05:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1141', diff saved to https://phabricator.wikimedia.org/P12102 and previous config saved to /var/cache/conftool/dbconfig/20200729-050247-marostegui.json |
[production] |
| 05:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1142', diff saved to https://phabricator.wikimedia.org/P12101 and previous config saved to /var/cache/conftool/dbconfig/20200729-050204-marostegui.json |
[production] |
| 04:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1142', diff saved to https://phabricator.wikimedia.org/P12100 and previous config saved to /var/cache/conftool/dbconfig/20200729-045859-marostegui.json |
[production] |
| 02:19 |
<tstarling@deploy1001> |
Synchronized wmf-config/CommonSettings.php: re-enable lilypond in safe mode (duration: 01m 09s) |
[production] |
| 01:47 |
<tstarling@deploy1001> |
Synchronized php-1.36.0-wmf.2/extensions/Score/includes/Score.php: work around firejail bug (duration: 01m 07s) |
[production] |
| 01:45 |
<tstarling@deploy1001> |
Synchronized php-1.36.0-wmf.1/extensions/Score/includes/Score.php: work around firejail bug (duration: 01m 08s) |
[production] |
| 01:19 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=wtp1048.eqiad.wmnet |
[production] |
| 01:15 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=wtp1047.eqiad.wmnet |
[production] |
| 00:53 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=wtp1046.eqiad.wmnet |
[production] |
| 00:48 |
<ryankemper> |
sudo -E cumin -b 10 'A:wdqs-all' 'sudo run-puppet-agent' |
[production] |
| 00:18 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 00:14 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
|
2020-07-28
§
|
| 23:38 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 23:37 |
<ebernhardson@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: cirrus: reduce mlr window size on enwiki (duration: 01m 05s) |
[production] |
| 23:36 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 23:34 |
<ebernhardson@deploy1001> |
Synchronized wmf-config/CirrusSearch-common.php: cirrus: reduce mlr window size on enwiki (duration: 01m 06s) |
[production] |
| 23:31 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 23:29 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 23:04 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Remove unused setting $wgGEHomepageSuggestedEditsNewAccountInitiatedPercentage (no-op) (duration: 01m 06s) |
[production] |
| 22:55 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=wtp1046.eqiad.wmnet |
[production] |
| 22:19 |
<jayme@cumin1001> |
conftool action : set/pooled=yes; selector: name=wtp1044.eqiad.wmnet |
[production] |
| 21:27 |
<dancy@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 21:24 |
<dancy@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 21:17 |
<dancy@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . |
[production] |
| 20:38 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 20:36 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:02 |
<eileen> |
process-control config revision is b6ece03513 |
[production] |
| 19:50 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 19:48 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 19:48 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 19:48 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |