|
2020-02-13
ยง
|
| 06:19 |
<vgutierrez> |
testing a new build of ATS 8.0.6 in cp40[26,32] |
[production] |
| 06:19 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1099:3318, db1099:3311 T239453', diff saved to https://phabricator.wikimedia.org/P10396 and previous config saved to /var/cache/conftool/dbconfig/20200213-061219-marostegui.json |
[production] |
| 06:11 |
<vgutierrez> |
depool cp10[89,90] and reimage as buster - T242093 |
[production] |
| 06:04 |
<vgutierrez> |
depool cp20[02,05] and reimage as buster - T242093 |
[production] |
| 06:03 |
<vgutierrez> |
pool cp20[01,08] running buster - T242093 |
[production] |
| 06:02 |
<twentyafterfour> |
set phabricator read-only to false |
[production] |
| 06:01 |
<twentyafterfour> |
set phabricator read-only |
[production] |
| 06:00 |
<marostegui> |
Start phabricator maintenance T244566 |
[production] |
| 05:55 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 05:53 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 05:53 |
<marostegui> |
Upgrade db1128 without restarting mysql - T244566 |
[production] |
| 05:52 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 05:50 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 05:47 |
<marostegui> |
Silence m3 hosts for maintenance - T244566 |
[production] |
| 05:38 |
<vgutierrez> |
depool cp2008 and reimage as buster - T242093 |
[production] |
| 05:37 |
<vgutierrez> |
pool cp2011 running buster - T242093 |
[production] |
| 05:35 |
<vgutierrez> |
depool cp2001 and reimage as buster - T242093 |
[production] |
| 05:34 |
<vgutierrez> |
pool cp2004 running buster - T242093 |
[production] |
| 05:30 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 05:28 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 05:28 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 05:25 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 05:09 |
<vgutierrez> |
depool cp20[04,11] and reimage as buster - T242093 |
[production] |
| 03:57 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 03:57 |
<pt1979@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 03:54 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 03:52 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 03:52 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 03:52 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 03:32 |
<pt1979@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 03:30 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 03:28 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 03:27 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 03:06 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 03:04 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 03:02 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 03:00 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 02:44 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 02:41 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 02:35 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 02:33 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 01:22 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 01:20 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 01:18 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 01:17 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 01:10 |
<twentyafterfour> |
no apparent problems with phabricator upgrade, all done |
[production] |
| 01:01 |
<twentyafterfour> |
starting phabricator deploy, momentary downtime expected while apache restarts |
[production] |
| 00:58 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 00:56 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |