|
2019-05-21
ยง
|
| 17:31 |
<bblack> |
eqiad LVS: low-traffic (all internal services): puppeting lvs1016, bringing back pybal in "secondary" role for all 3 traffic classes (high-traffic1, high-traffic2, low-traffic), no traffic shift expected (again, after merging last-minute fixup https://gerrit.wikimedia.org/r/c/operations/puppet/+/511759 ) |
[production] |
| 17:25 |
<bblack> |
eqiad LVS: low-traffic (all internal services): puppeting lvs1016, bringing back pybal in "secondary" role for all 3 traffic classes (high-traffic1, high-traffic2, low-traffic), no traffic shift expected |
[production] |
| 17:24 |
<bblack> |
eqiad LVS: low-traffic (all internal services): puppeting lvs1006, basically no-op |
[production] |
| 17:21 |
<bblack> |
eqiad LVS: low-traffic (all internal services): puppeting lvs1015, bringing back pybal in primary role, shifting traffic to lvs1015 |
[production] |
| 17:20 |
<bblack> |
eqiad LVS: low-traffic (all internal services): disable pybal on lvs1016 + lvs1015, shifting traffic to lvs1006 |
[production] |
| 17:18 |
<reedy@deploy1001> |
Synchronized php-1.34.0-wmf.6/extensions/Collection/includes/CollectionHooks.php: Fix paths (duration: 00m 56s) |
[production] |
| 17:17 |
<bblack> |
eqiad LVS: high-traffic2 (upload): puppeting lvs1005, basically no-op |
[production] |
| 17:15 |
<bblack> |
eqiad LVS: high-traffic2 (upload): puppeting lvs1002, bringing back pybal in backup role, no traffic shift |
[production] |
| 17:13 |
<bblack> |
eqiad LVS: high-traffic2 (upload): puppeting lvs1014, bringing back pybal in primary role, shifting traffic to lvs1014 |
[production] |
| 17:11 |
<bblack> |
eqiad LVS: high-traffic2 (upload): disable pybal on lvs1014 + lvs1002, shifting traffic to lvs1005 |
[production] |
| 17:09 |
<bblack> |
eqiad LVS: high-traffic1 (text): puppeting lvs1004, basically no-op |
[production] |
| 17:07 |
<bblack> |
eqiad LVS: high-traffic1 (text): puppeting lvs1001, bringing back pybal in backup role, no traffic shift |
[production] |
| 17:06 |
<bblack> |
eqiad LVS: high-traffic1 (text): puppeting lvs1013, bringing back pybal in primary role, shifting traffic to lvs1013 |
[production] |
| 17:04 |
<bblack> |
eqiad LVS: high-traffic1 (text): disable pybal on lvs1013 + lvs1001, shifting traffic to lvs1004 |
[production] |
| 16:55 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:55 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:55 |
<jbond42> |
rebooting wtp1046-1048 |
[production] |
| 16:55 |
<bblack> |
starting Eqiad LVS re-arrangement shortly - T184293 - https://gerrit.wikimedia.org/r/c/operations/puppet/+/511717 (eqiad front edge is still depooled from public traffic) |
[production] |
| 16:50 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:50 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:50 |
<jbond42> |
rebooting wtp1043-1045 |
[production] |
| 16:46 |
<mutante> |
rebooting phab1003 (non-prod) |
[production] |
| 16:44 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:44 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:44 |
<jbond42> |
rebooting wtp1040-1042 |
[production] |
| 16:39 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:39 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:39 |
<jbond42> |
rebooting wtp1037-1039 |
[production] |
| 16:26 |
<mobrovac> |
truncate "others_T_parsoid".data |
[production] |
| 16:25 |
<mobrovac> |
restbase truncate "commons_T_parsoid".data |
[production] |
| 16:24 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:24 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:24 |
<jbond42> |
rebooting wtp1033-1034 |
[production] |
| 16:18 |
<mobrovac> |
restbase truncate "enwiki_T_parsoid".data |
[production] |
| 16:16 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:16 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:16 |
<jbond42> |
rebooting wtp1031-1032 |
[production] |
| 16:10 |
<mobrovac> |
restbase truncate "wikipedia_T_parsoid".data |
[production] |
| 16:09 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:09 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:09 |
<jbond42> |
rebooting wtp1029-2030 |
[production] |
| 16:01 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:01 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:01 |
<jbond42> |
rebooting wtp1027-2028 |
[production] |
| 15:56 |
<urandom> |
decommissioning restbase1007-a -- T208087 |
[production] |
| 15:54 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:54 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:54 |
<jbond42> |
rebooting wtp1025-2026 |
[production] |
| 15:45 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@cf00120]: Revert Temporarily copy from old tables to new ones if the data is not found, rb1007 (duration: 02m 43s) |
[production] |
| 15:42 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@cf00120]: Revert Temporarily copy from old tables to new ones if the data is not found, rb1007 |
[production] |