|
2019-09-25
§
|
| 07:17 |
<onimisionipe> |
pool wdqs1005 to allow depooling wdqs1004 to handle lag issues |
[production] |
| 07:17 |
<elukey> |
allow analytics users to log in into stat1005 |
[production] |
| 06:33 |
<_joe_> |
restarting pybal on all low-traffic lbs |
[production] |
| 06:29 |
<@> |
helmfile [CODFW] Ran 'sync' command on namespace 'restrouter' for release 'codfw' . |
[production] |
| 06:29 |
<@> |
helmfile [EQIAD] Ran 'sync' command on namespace 'restrouter' for release 'production' . |
[production] |
| 06:21 |
<marostegui> |
Deploy schema change on db2085:3311 T233625 |
[production] |
| 06:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2085:3311 T233625', diff saved to https://phabricator.wikimedia.org/P9171 and previous config saved to /var/cache/conftool/dbconfig/20190925-062036-marostegui.json |
[production] |
| 05:33 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=False) |
[production] |
| 05:33 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 05:24 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=False) |
[production] |
| 05:24 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 05:24 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=False) |
[production] |
| 05:23 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 05:23 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=False) |
[production] |
| 05:23 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 05:23 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=False) |
[production] |
| 05:23 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 05:11 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=False) |
[production] |
| 05:11 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 05:06 |
<marostegui> |
Run a data check on labsdb1011 - T233766 |
[production] |
| 04:43 |
<marostegui> |
Deploy schema change on s3 with replication - T231172 |
[production] |
| 03:28 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.34.0-wmf.24 refs T220749 |
[production] |
| 03:03 |
<krinkle@deploy1001> |
Synchronized docroot/noc/: c7c6c0ee0, 8405bf1c2 (duration: 01m 05s) |
[production] |
| 03:01 |
<krinkle@deploy1001> |
Synchronized src/: c7c6c0ee0, 8405bf1c2 (for noc.wm.o) (duration: 01m 09s) |
[production] |
| 02:58 |
<twentyafterfour> |
belatedly promoting wmf.24 to group0 refs T220749 |
[production] |
| 02:32 |
<onimisionipe> |
depool wdqs1005 to let it catch up with lag |
[production] |
| 02:30 |
<onimisionipe> |
pool wdqs1006 - it has caught up with lag |
[production] |
| 01:16 |
<mutante> |
stat1007 - restart nagios-nrpe-server, echo "please don't use all of the RAM on this server" | wall |
[production] |
| 01:14 |
<krinkle@deploy1001> |
Synchronized wmf-config/: 3373247e12 (duration: 01m 04s) |
[production] |
| 01:12 |
<krinkle@deploy1001> |
Synchronized src/WmfClusters.php: 3373247e123b (duration: 01m 04s) |
[production] |
| 01:08 |
<krinkle@deploy1001> |
Synchronized tests: 3373247e123b5 (duration: 01m 04s) |
[production] |
| 01:07 |
<krinkle@deploy1001> |
Synchronized docroot/noc: 3373247e123b53 and 1efc8bd68107877311a749 (duration: 01m 05s) |
[production] |
| 01:03 |
<krinkle@deploy1001> |
Synchronized README: 3373247e123b53 (duration: 01m 04s) |
[production] |
| 01:00 |
<krinkle@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 3373247e123b53 - create new file (duration: 01m 05s) |
[production] |
| 00:47 |
<krinkle@deploy1001> |
Synchronized wmf-config/: 6dca83a9f6c2c (duration: 01m 04s) |
[production] |
| 00:44 |
<krinkle@deploy1001> |
Synchronized docroot/noc/: 6dca83a9f6c2c (duration: 01m 05s) |
[production] |
| 00:43 |
<krinkle@deploy1001> |
Synchronized tests/: 6dca83a9f6c2c (duration: 01m 05s) |
[production] |
| 00:02 |
<mutante> |
cp1075 - systemctl restart vhtcpd |
[production] |
| 00:02 |
<mutante> |
cp1075 - systemctl status vhtcpd |
[production] |
|
2019-09-24
§
|
| 23:38 |
<mutante> |
gerrit service restart to switch LDAP backend |
[production] |
| 23:35 |
<bstorm_> |
wiki-replicas depooled labsdb1011 |
[production] |
| 23:33 |
<mutante> |
gerrit2001 - restarting gerrit service |
[production] |
| 23:30 |
<mutante> |
switching LDAP servers used by Gerrit to readonly replicas. stop using so called "labs" config for LDAP backend. |
[production] |
| 22:25 |
<twentyafterfour@deploy1001> |
Finished scap: testwikis wikis to 1.34.0-wmf.24 refs T220749 (duration: 40m 38s) |
[production] |
| 21:53 |
<mutante> |
restbase1024 - enable IPMI over LAN which wasn't working before |
[production] |
| 21:45 |
<twentyafterfour@deploy1001> |
Started scap: testwikis wikis to 1.34.0-wmf.24 refs T220749 |
[production] |
| 21:19 |
<mutante> |
ganeti4001 - racadm racreset - attempt to fix IPMI |
[production] |
| 20:19 |
<twentyafterfour> |
restarting gerrit due to unreasonably high garbage collection times and sluggish performance in general. |
[production] |
| 19:39 |
<XioNoX> |
disable asw2-d-eqiad:ge-5/0/41 excessive flapping |
[production] |
| 19:28 |
<ejegg> |
updated payments-wiki from 939b771800 to 5193dcdfa9 |
[production] |