|
2019-05-21
ยง
|
| 17:17 |
<bblack> |
eqiad LVS: high-traffic2 (upload): puppeting lvs1005, basically no-op |
[production] |
| 17:15 |
<bblack> |
eqiad LVS: high-traffic2 (upload): puppeting lvs1002, bringing back pybal in backup role, no traffic shift |
[production] |
| 17:13 |
<bblack> |
eqiad LVS: high-traffic2 (upload): puppeting lvs1014, bringing back pybal in primary role, shifting traffic to lvs1014 |
[production] |
| 17:11 |
<bblack> |
eqiad LVS: high-traffic2 (upload): disable pybal on lvs1014 + lvs1002, shifting traffic to lvs1005 |
[production] |
| 17:09 |
<bblack> |
eqiad LVS: high-traffic1 (text): puppeting lvs1004, basically no-op |
[production] |
| 17:07 |
<bblack> |
eqiad LVS: high-traffic1 (text): puppeting lvs1001, bringing back pybal in backup role, no traffic shift |
[production] |
| 17:06 |
<bblack> |
eqiad LVS: high-traffic1 (text): puppeting lvs1013, bringing back pybal in primary role, shifting traffic to lvs1013 |
[production] |
| 17:04 |
<bblack> |
eqiad LVS: high-traffic1 (text): disable pybal on lvs1013 + lvs1001, shifting traffic to lvs1004 |
[production] |
| 16:55 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:55 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:55 |
<jbond42> |
rebooting wtp1046-1048 |
[production] |
| 16:55 |
<bblack> |
starting Eqiad LVS re-arrangement shortly - T184293 - https://gerrit.wikimedia.org/r/c/operations/puppet/+/511717 (eqiad front edge is still depooled from public traffic) |
[production] |
| 16:50 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:50 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:50 |
<jbond42> |
rebooting wtp1043-1045 |
[production] |
| 16:46 |
<mutante> |
rebooting phab1003 (non-prod) |
[production] |
| 16:44 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:44 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:44 |
<jbond42> |
rebooting wtp1040-1042 |
[production] |
| 16:39 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:39 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:39 |
<jbond42> |
rebooting wtp1037-1039 |
[production] |
| 16:26 |
<mobrovac> |
truncate "others_T_parsoid".data |
[production] |
| 16:25 |
<mobrovac> |
restbase truncate "commons_T_parsoid".data |
[production] |
| 16:24 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:24 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:24 |
<jbond42> |
rebooting wtp1033-1034 |
[production] |
| 16:18 |
<mobrovac> |
restbase truncate "enwiki_T_parsoid".data |
[production] |
| 16:16 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:16 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:16 |
<jbond42> |
rebooting wtp1031-1032 |
[production] |
| 16:10 |
<mobrovac> |
restbase truncate "wikipedia_T_parsoid".data |
[production] |
| 16:09 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:09 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:09 |
<jbond42> |
rebooting wtp1029-2030 |
[production] |
| 16:01 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:01 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:01 |
<jbond42> |
rebooting wtp1027-2028 |
[production] |
| 15:56 |
<urandom> |
decommissioning restbase1007-a -- T208087 |
[production] |
| 15:54 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:54 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:54 |
<jbond42> |
rebooting wtp1025-2026 |
[production] |
| 15:45 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@cf00120]: Revert Temporarily copy from old tables to new ones if the data is not found, rb1007 (duration: 02m 43s) |
[production] |
| 15:42 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@cf00120]: Revert Temporarily copy from old tables to new ones if the data is not found, rb1007 |
[production] |
| 15:42 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@cf00120]: Revert Temporarily copy from old tables to new ones if the data is not found (duration: 02m 40s) |
[production] |
| 15:40 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:40 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:40 |
<jbond42> |
rebooting wtp2019-2020 |
[production] |
| 15:39 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@cf00120]: Revert Temporarily copy from old tables to new ones if the data is not found |
[production] |
| 15:38 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@022cb98]: Temporarily copy from old tables to new ones if the data is not found, take #2 (duration: 00m 45s) |
[production] |