|
2020-08-13
§
|
| 07:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1082', diff saved to https://phabricator.wikimedia.org/P12244 and previous config saved to /var/cache/conftool/dbconfig/20200813-071943-marostegui.json |
[production] |
| 07:16 |
<marostegui> |
Stop replication on db1082 to remove triggers on sanitarium for MCR changs |
[production] |
| 07:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1082', diff saved to https://phabricator.wikimedia.org/P12243 and previous config saved to /var/cache/conftool/dbconfig/20200813-071545-marostegui.json |
[production] |
| 06:48 |
<marostegui> |
Deploy MCR change on dbstore1003:3311 |
[production] |
| 06:02 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1126', diff saved to https://phabricator.wikimedia.org/P12242 and previous config saved to /var/cache/conftool/dbconfig/20200813-060135-marostegui.json |
[production] |
| 06:00 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 05:43 |
<marostegui> |
Stop MySQL on db2135 (codfw master), haproxy irc alert will fire T260324 |
[production] |
| 05:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1126', diff saved to https://phabricator.wikimedia.org/P12241 and previous config saved to /var/cache/conftool/dbconfig/20200813-052859-marostegui.json |
[production] |
| 05:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1126', diff saved to https://phabricator.wikimedia.org/P12240 and previous config saved to /var/cache/conftool/dbconfig/20200813-051222-marostegui.json |
[production] |
| 05:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1126', diff saved to https://phabricator.wikimedia.org/P12239 and previous config saved to /var/cache/conftool/dbconfig/20200813-050107-marostegui.json |
[production] |
| 02:56 |
<mutante> |
testreduce1001 - systemctl reset-failed ; fix parsoid-vd systemd state and icinga alert |
[production] |
| 00:37 |
<mutante> |
removing jenkins_service_running checks from secondary servers where it's stopped, manually from icinga config, running puppet on icinga |
[production] |
| 00:14 |
<mutante> |
re-enabling puppet on releases* servers |
[production] |
|
2020-08-12
§
|
| 23:44 |
<wkandek@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 23:41 |
<wkandek@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 23:40 |
<wkandek@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 23:39 |
<wkandek@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 23:37 |
<wkandek> |
reboot mw1372 |
[production] |
| 23:36 |
<wkandek@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 23:36 |
<wkandek@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 23:32 |
<wkandek> |
reboot mw1373 |
[production] |
| 23:32 |
<wkandek@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 23:31 |
<wkandek> |
reboot mw1371 |
[production] |
| 23:31 |
<wkandek@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 23:31 |
<wkandek@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 23:30 |
<wkandek@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 23:28 |
<wkandek> |
reboot mw1384 |
[production] |
| 23:27 |
<wkandek@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 23:27 |
<wkandek> |
reboot mw1385 |
[production] |
| 23:26 |
<wkandek@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 23:25 |
<wkandek@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 23:24 |
<wkandek@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 23:22 |
<wkandek> |
reboot mw1370 |
[production] |
| 23:22 |
<wkandek@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 23:19 |
<wkandek@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 23:18 |
<wkandek> |
reboot mw1369 |
[production] |
| 23:18 |
<wkandek@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 23:17 |
<wkandek@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 23:17 |
<wkandek> |
reboot mw1387 |
[production] |
| 23:16 |
<wkandek@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 23:16 |
<wkandek@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 23:16 |
<wkandek> |
reboot mw1389 |
[production] |
| 23:15 |
<wkandek@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 23:14 |
<wkandek@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 23:09 |
<wkandek> |
reboot mw1368 |
[production] |
| 23:09 |
<wkandek@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 23:08 |
<wkandek> |
reboot me1367 |
[production] |
| 23:08 |
<wkandek@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 23:07 |
<wkandek> |
reboot mw1391 |
[production] |