|
2020-07-13
§
|
| 12:08 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Start repooling es1022 after reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11873 and previous config saved to /var/cache/conftool/dbconfig/20200713-120818-kormat.json |
[production] |
| 11:49 |
<Urbanecm> |
Password reset for User:Alert5 (T257806) |
[production] |
| 11:44 |
<akosiaris> |
repool ganeti1007 T244530. Start emptying ganeti1008 |
[production] |
| 11:08 |
<Urbanecm> |
EU B&C done |
[production] |
| 11:06 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 896c042296b4e1f5d88f786981537655e5d9fea9: Enable SandboxLink extension in trwiki (T256782) (duration: 00m 56s) |
[production] |
| 10:44 |
<jdrewniak@deploy1001> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:612175| Bumping portals to master (612175)]] (duration: 00m 56s) |
[production] |
| 10:43 |
<jdrewniak@deploy1001> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:612175| Bumping portals to master (612175)]] (duration: 00m 56s) |
[production] |
| 09:42 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:39 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:58 |
<ema> |
cp: rolling ats-backend-restart to apply SyslogIdentifier changes -> https://gerrit.wikimedia.org/r/c/operations/puppet/+/611311 |
[production] |
| 08:57 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: T248343 Explicitly set visualeditor-enable to 0 when non-default (duration: 00m 57s) |
[production] |
| 08:44 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Depool es1022 for reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11871 and previous config saved to /var/cache/conftool/dbconfig/20200713-084449-kormat.json |
[production] |
| 08:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1093', diff saved to https://phabricator.wikimedia.org/P11870 and previous config saved to /var/cache/conftool/dbconfig/20200713-083902-marostegui.json |
[production] |
| 08:34 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Add weight to es1020, reduce weight on es1022 T257284', diff saved to https://phabricator.wikimedia.org/P11869 and previous config saved to /var/cache/conftool/dbconfig/20200713-083414-kormat.json |
[production] |
| 08:20 |
<kormat> |
reimaging es1022 T257284 |
[production] |
| 06:54 |
<marostegui@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 06:53 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 06:52 |
<marostegui@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 06:52 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 06:51 |
<marostegui@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) |
[production] |
| 06:50 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 06:16 |
<marostegui> |
Reverse gerrit password on m2 master - T255715 |
[production] |
| 06:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1093', diff saved to https://phabricator.wikimedia.org/P11868 and previous config saved to /var/cache/conftool/dbconfig/20200713-060410-marostegui.json |
[production] |
| 05:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1093', diff saved to https://phabricator.wikimedia.org/P11867 and previous config saved to /var/cache/conftool/dbconfig/20200713-055422-marostegui.json |
[production] |
| 05:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1093 for upgrade', diff saved to https://phabricator.wikimedia.org/P11866 and previous config saved to /var/cache/conftool/dbconfig/20200713-054840-marostegui.json |
[production] |
| 05:34 |
<marostegui> |
Deploy schema change on s3 codfw master, lag will appear on codfw T253276 |
[production] |
| 05:30 |
<marostegui> |
Stop replication on db1082 for schema change and triggers removal T238966 |
[production] |
| 05:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1082', diff saved to https://phabricator.wikimedia.org/P11865 and previous config saved to /var/cache/conftool/dbconfig/20200713-052928-marostegui.json |
[production] |
| 05:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1119 for innodb compression', diff saved to https://phabricator.wikimedia.org/P11864 and previous config saved to /var/cache/conftool/dbconfig/20200713-051428-marostegui.json |
[production] |
|
2020-07-10
§
|
| 21:52 |
<ryankemper> |
Started long-running reindex of Elasticsearch indices in `eqiad`, `codfw`, and `dewiki` on `mwmaint1002` under tmux session `reindex` for user `ryankemper` |
[production] |
| 20:26 |
<jgleeson> |
updated fundraising-tools from 08ba1f6177 to f8e424fe32 |
[production] |
| 19:02 |
<mutante> |
removing firewall hole for gerrit -> mysql servers on dbproxy servers for misc db's |
[production] |
| 18:44 |
<mutante> |
kubernetes1004 - started nagios-nrpe-server |
[production] |
| 17:57 |
<ebernhardson> |
change loginwiki password for Cindy-the-browser-test-bot, no email account was associated to allow for normal reset. |
[production] |
| 17:05 |
<krinkle@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: I63fcea7737 (duration: 00m 57s) |
[production] |
| 16:16 |
<elukey@cumin1001> |
END (FAIL) - Cookbook sre.hadoop.change-distro (exit_code=99) |
[production] |
| 15:57 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@4d40145] (thin): Update EventLogging refine whitelist (THIN) (duration: 00m 08s) |
[production] |
| 15:56 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@4d40145] (thin): Update EventLogging refine whitelist (THIN) |
[production] |
| 15:44 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@4d40145]: Update EventLogging refine whitelist (duration: 15m 17s) |
[production] |
| 15:30 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.change-distro |
[production] |
| 15:29 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.stop-cluster (exit_code=0) |
[production] |
| 15:29 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@4d40145]: Update EventLogging refine whitelist |
[production] |
| 15:19 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.stop-cluster |
[production] |
| 15:03 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.change-distro (exit_code=0) |
[production] |