|
2020-11-03
§
|
| 11:03 |
<Amir1> |
rolling restart of ores |
[production] |
| 10:59 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:59 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:45 |
<gilles@deploy1001> |
Finished deploy [performance/asoranking@2a2cb05]: T266985 (duration: 00m 07s) |
[production] |
| 10:45 |
<gilles@deploy1001> |
Started deploy [performance/asoranking@2a2cb05]: T266985 |
[production] |
| 10:39 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:39 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:22 |
<gilles@deploy1001> |
Finished deploy [performance/asoranking@2a2cb05]: T266985 (duration: 00m 26s) |
[production] |
| 10:21 |
<gilles@deploy1001> |
Started deploy [performance/asoranking@2a2cb05]: T266985 |
[production] |
| 10:16 |
<elukey@deploy1001> |
Finished deploy [analytics/refinery@cf5db74] (hadoop-test): (no justification provided) (duration: 02m 15s) |
[production] |
| 10:14 |
<elukey@deploy1001> |
Started deploy [analytics/refinery@cf5db74] (hadoop-test): (no justification provided) |
[production] |
| 10:13 |
<elukey@deploy1001> |
Finished deploy [analytics/refinery@cf5db74] (hadoop-test): (no justification provided) (duration: 01m 45s) |
[production] |
| 10:11 |
<elukey@deploy1001> |
Started deploy [analytics/refinery@cf5db74] (hadoop-test): (no justification provided) |
[production] |
| 10:08 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:08 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:57 |
<kormat> |
uploaded orchestrator 3.2.3-2 to apt |
[production] |
| 09:54 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:54 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:49 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:49 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:18 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:18 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:12 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:12 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:06 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:06 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:05 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Repooling after reboot. T261389', diff saved to https://phabricator.wikimedia.org/P13139 and previous config saved to /var/cache/conftool/dbconfig/20201103-090523-kormat.json |
[production] |
| 09:00 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Rebooting for T261389', diff saved to https://phabricator.wikimedia.org/P13138 and previous config saved to /var/cache/conftool/dbconfig/20201103-090013-kormat.json |
[production] |
| 09:00 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:00 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:32 |
<godog> |
Prometheus re-enable compactions - T261281 |
[production] |
| 06:59 |
<marostegui> |
Remove db1091 from tendril and zarcillo T267088 |
[production] |
| 06:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1091 from dbctl T267088', diff saved to https://phabricator.wikimedia.org/P13137 and previous config saved to /var/cache/conftool/dbconfig/20201103-065756-marostegui.json |
[production] |
| 06:46 |
<marostegui> |
Deploy schema change on s1 codfw master: T265349 |
[production] |
| 06:16 |
<marostegui> |
Stop MySQL on es1014 to clone es1028 T261717 |
[production] |
| 06:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1014 to reclone es1028 T261717', diff saved to https://phabricator.wikimedia.org/P13136 and previous config saved to /var/cache/conftool/dbconfig/20201103-061423-marostegui.json |
[production] |
| 06:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote es1019 to es3 master (this is a noop) T261717', diff saved to https://phabricator.wikimedia.org/P13135 and previous config saved to /var/cache/conftool/dbconfig/20201103-061403-marostegui.json |
[production] |
| 06:11 |
<marostegui> |
Stop MySQL on es1012 to clone es1027 T261717 |
[production] |
| 06:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1012 to reclone es1027 T261717', diff saved to https://phabricator.wikimedia.org/P13134 and previous config saved to /var/cache/conftool/dbconfig/20201103-060727-marostegui.json |
[production] |
| 06:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote es1018 to es1 master (this is a noop) T261717', diff saved to https://phabricator.wikimedia.org/P13133 and previous config saved to /var/cache/conftool/dbconfig/20201103-060705-marostegui.json |
[production] |
| 06:04 |
<marostegui> |
Stop MySQL on es1011 to clone es1026 T261717 |
[production] |
| 06:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1011 to reclone es1026 T261717', diff saved to https://phabricator.wikimedia.org/P13132 and previous config saved to /var/cache/conftool/dbconfig/20201103-060054-marostegui.json |
[production] |
| 06:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote es1015 to es2 master (this is a noop) T261717', diff saved to https://phabricator.wikimedia.org/P13131 and previous config saved to /var/cache/conftool/dbconfig/20201103-060038-marostegui.json |
[production] |
| 04:39 |
<cstone> |
civicrm revision changed from cd13d9e30f to b1342c4129 |
[production] |
| 02:13 |
<shdubsh> |
restart ES on logstash1009 - oom killed |
[production] |
| 01:01 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 00:59 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 00:42 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 00:40 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |