|
2019-05-21
ยง
|
| 23:34 |
<maxsem@deploy1001> |
Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/511667/ (duration: 00m 56s) |
[production] |
| 22:56 |
<mutante> |
ms-be2034 - degraded systemd state was cleared and originally caused by " failed Session 72587 of user debmonitor" |
[production] |
| 22:56 |
<mutante> |
ms-be2034 - sudo systemctl reset-failed |
[production] |
| 22:51 |
<urandom> |
decommissioning restbase1007-b -- T223976 |
[production] |
| 21:35 |
<ejegg> |
updated payments-wiki from d5ef5ad067 to fa005a0640 |
[production] |
| 21:21 |
<mutante> |
re-enabling puppet on mc1* hosts |
[production] |
| 20:43 |
<mutante> |
re-enabling puppet on all hosts using memcached class - except mc1* |
[production] |
| 20:31 |
<mutante> |
mc2019 - stopping memcached and letting puppet restart it to confirm no issues after switching to systemd::service |
[production] |
| 20:20 |
<mutante> |
disabling puppet on all servers using class memcached (57) |
[production] |
| 20:06 |
<tzatziki> |
removing (another) two files for legal compliance |
[production] |
| 19:43 |
<tzatziki> |
removing two files for legal compliance |
[production] |
| 19:12 |
<thcipriani> |
gerrit back on 2.15.13 |
[production] |
| 19:09 |
<thcipriani> |
restart gerrit for 2.15.13 update |
[production] |
| 19:08 |
<thcipriani@deploy1001> |
Finished deploy [gerrit/gerrit@2de9001]: Gerrit to 2.15.13 (cobalt, restart incoming) (duration: 00m 20s) |
[production] |
| 19:08 |
<thcipriani@deploy1001> |
Started deploy [gerrit/gerrit@2de9001]: Gerrit to 2.15.13 (cobalt, restart incoming) |
[production] |
| 19:06 |
<thcipriani@deploy1001> |
Finished deploy [gerrit/gerrit@2de9001]: Gerrit to 2.15.13 (gerrit 2001 only) (duration: 00m 11s) |
[production] |
| 19:06 |
<thcipriani@deploy1001> |
Started deploy [gerrit/gerrit@2de9001]: Gerrit to 2.15.13 (gerrit 2001 only) |
[production] |
| 18:50 |
<bblack> |
repooling cp1085 frontends (weren't meant to be depooled) |
[production] |
| 18:38 |
<bblack> |
re-pooling eqiad front edge traffic (onto new LVSes from T184293 ) |
[production] |
| 18:35 |
<XioNoX> |
update lvs static routes on cr1/2-eqiad - T184293 |
[production] |
| 18:06 |
<andrewbogott> |
restarting rabbitmq-server on cloudcontrol1003 (turning on HA queues) |
[production] |
| 17:59 |
<bblack> |
rebooting lvs1016 in attempt to clear interface config issues - T224027 |
[production] |
| 17:51 |
<XioNoX> |
add BGP sessions to AS202053 in esams |
[production] |
| 17:31 |
<bblack> |
eqiad LVS: low-traffic (all internal services): puppeting lvs1016, bringing back pybal in "secondary" role for all 3 traffic classes (high-traffic1, high-traffic2, low-traffic), no traffic shift expected (again, after merging last-minute fixup https://gerrit.wikimedia.org/r/c/operations/puppet/+/511759 ) |
[production] |
| 17:25 |
<bblack> |
eqiad LVS: low-traffic (all internal services): puppeting lvs1016, bringing back pybal in "secondary" role for all 3 traffic classes (high-traffic1, high-traffic2, low-traffic), no traffic shift expected |
[production] |
| 17:24 |
<bblack> |
eqiad LVS: low-traffic (all internal services): puppeting lvs1006, basically no-op |
[production] |
| 17:21 |
<bblack> |
eqiad LVS: low-traffic (all internal services): puppeting lvs1015, bringing back pybal in primary role, shifting traffic to lvs1015 |
[production] |
| 17:20 |
<bblack> |
eqiad LVS: low-traffic (all internal services): disable pybal on lvs1016 + lvs1015, shifting traffic to lvs1006 |
[production] |
| 17:18 |
<reedy@deploy1001> |
Synchronized php-1.34.0-wmf.6/extensions/Collection/includes/CollectionHooks.php: Fix paths (duration: 00m 56s) |
[production] |
| 17:17 |
<bblack> |
eqiad LVS: high-traffic2 (upload): puppeting lvs1005, basically no-op |
[production] |
| 17:15 |
<bblack> |
eqiad LVS: high-traffic2 (upload): puppeting lvs1002, bringing back pybal in backup role, no traffic shift |
[production] |
| 17:13 |
<bblack> |
eqiad LVS: high-traffic2 (upload): puppeting lvs1014, bringing back pybal in primary role, shifting traffic to lvs1014 |
[production] |
| 17:11 |
<bblack> |
eqiad LVS: high-traffic2 (upload): disable pybal on lvs1014 + lvs1002, shifting traffic to lvs1005 |
[production] |
| 17:09 |
<bblack> |
eqiad LVS: high-traffic1 (text): puppeting lvs1004, basically no-op |
[production] |
| 17:07 |
<bblack> |
eqiad LVS: high-traffic1 (text): puppeting lvs1001, bringing back pybal in backup role, no traffic shift |
[production] |
| 17:06 |
<bblack> |
eqiad LVS: high-traffic1 (text): puppeting lvs1013, bringing back pybal in primary role, shifting traffic to lvs1013 |
[production] |
| 17:04 |
<bblack> |
eqiad LVS: high-traffic1 (text): disable pybal on lvs1013 + lvs1001, shifting traffic to lvs1004 |
[production] |
| 16:55 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:55 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:55 |
<jbond42> |
rebooting wtp1046-1048 |
[production] |
| 16:55 |
<bblack> |
starting Eqiad LVS re-arrangement shortly - T184293 - https://gerrit.wikimedia.org/r/c/operations/puppet/+/511717 (eqiad front edge is still depooled from public traffic) |
[production] |
| 16:50 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:50 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:50 |
<jbond42> |
rebooting wtp1043-1045 |
[production] |
| 16:46 |
<mutante> |
rebooting phab1003 (non-prod) |
[production] |
| 16:44 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:44 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:44 |
<jbond42> |
rebooting wtp1040-1042 |
[production] |
| 16:39 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:39 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |