|
2020-11-03
§
|
| 10:13 |
<elukey@deploy1001> |
Finished deploy [analytics/refinery@cf5db74] (hadoop-test): (no justification provided) (duration: 01m 45s) |
[production] |
| 10:11 |
<elukey@deploy1001> |
Started deploy [analytics/refinery@cf5db74] (hadoop-test): (no justification provided) |
[production] |
| 10:08 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:08 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:57 |
<kormat> |
uploaded orchestrator 3.2.3-2 to apt |
[production] |
| 09:54 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:54 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:49 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:49 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:18 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:18 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:12 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:12 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:06 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:06 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:05 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Repooling after reboot. T261389', diff saved to https://phabricator.wikimedia.org/P13139 and previous config saved to /var/cache/conftool/dbconfig/20201103-090523-kormat.json |
[production] |
| 09:00 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Rebooting for T261389', diff saved to https://phabricator.wikimedia.org/P13138 and previous config saved to /var/cache/conftool/dbconfig/20201103-090013-kormat.json |
[production] |
| 09:00 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:00 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:32 |
<godog> |
Prometheus re-enable compactions - T261281 |
[production] |
| 06:59 |
<marostegui> |
Remove db1091 from tendril and zarcillo T267088 |
[production] |
| 06:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1091 from dbctl T267088', diff saved to https://phabricator.wikimedia.org/P13137 and previous config saved to /var/cache/conftool/dbconfig/20201103-065756-marostegui.json |
[production] |
| 06:46 |
<marostegui> |
Deploy schema change on s1 codfw master: T265349 |
[production] |
| 06:16 |
<marostegui> |
Stop MySQL on es1014 to clone es1028 T261717 |
[production] |
| 06:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1014 to reclone es1028 T261717', diff saved to https://phabricator.wikimedia.org/P13136 and previous config saved to /var/cache/conftool/dbconfig/20201103-061423-marostegui.json |
[production] |
| 06:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote es1019 to es3 master (this is a noop) T261717', diff saved to https://phabricator.wikimedia.org/P13135 and previous config saved to /var/cache/conftool/dbconfig/20201103-061403-marostegui.json |
[production] |
| 06:11 |
<marostegui> |
Stop MySQL on es1012 to clone es1027 T261717 |
[production] |
| 06:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1012 to reclone es1027 T261717', diff saved to https://phabricator.wikimedia.org/P13134 and previous config saved to /var/cache/conftool/dbconfig/20201103-060727-marostegui.json |
[production] |
| 06:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote es1018 to es1 master (this is a noop) T261717', diff saved to https://phabricator.wikimedia.org/P13133 and previous config saved to /var/cache/conftool/dbconfig/20201103-060705-marostegui.json |
[production] |
| 06:04 |
<marostegui> |
Stop MySQL on es1011 to clone es1026 T261717 |
[production] |
| 06:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1011 to reclone es1026 T261717', diff saved to https://phabricator.wikimedia.org/P13132 and previous config saved to /var/cache/conftool/dbconfig/20201103-060054-marostegui.json |
[production] |
| 06:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote es1015 to es2 master (this is a noop) T261717', diff saved to https://phabricator.wikimedia.org/P13131 and previous config saved to /var/cache/conftool/dbconfig/20201103-060038-marostegui.json |
[production] |
| 04:39 |
<cstone> |
civicrm revision changed from cd13d9e30f to b1342c4129 |
[production] |
| 02:13 |
<shdubsh> |
restart ES on logstash1009 - oom killed |
[production] |
| 01:01 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 00:59 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 00:42 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 00:40 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
|
2020-11-02
§
|
| 22:19 |
<twentyafterfour> |
restart php7.3-fpm on phab1001 |
[production] |
| 22:03 |
<twentyafterfour> |
applied 113a244a66 on phab1001 to hotfix T240862 |
[production] |
| 20:22 |
<eileen> |
process-control config revision is 313a36312f re-enable thank you |
[production] |
| 19:56 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 19:48 |
<dzahn@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 19:47 |
<eileen> |
civicrm revision changed from 3317d30356 to cd13d9e30f, config revision is db912e3bba |
[production] |
| 19:45 |
<eileen> |
process-control config revision is db912e3bba - thankyou job off for testing |
[production] |
| 19:07 |
<Urbanecm> |
Deployed security fix for T205908 |
[production] |
| 19:04 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 18:59 |
<andrewbogott> |
added dcaro to ops and wmf ldap groups |
[production] |
| 18:59 |
<mutante> |
decom'ing testvm1001 |
[production] |
| 18:58 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |