4751-4800 of 10000 results (72ms)
2019-05-21 ยง
21:21 <mutante> re-enabling puppet on mc1* hosts [production]
20:43 <mutante> re-enabling puppet on all hosts using memcached class - except mc1* [production]
20:31 <mutante> mc2019 - stopping memcached and letting puppet restart it to confirm no issues after switching to systemd::service [production]
20:20 <mutante> disabling puppet on all servers using class memcached (57) [production]
20:06 <tzatziki> removing (another) two files for legal compliance [production]
19:43 <tzatziki> removing two files for legal compliance [production]
19:12 <thcipriani> gerrit back on 2.15.13 [production]
19:09 <thcipriani> restart gerrit for 2.15.13 update [production]
19:08 <thcipriani@deploy1001> Finished deploy [gerrit/gerrit@2de9001]: Gerrit to 2.15.13 (cobalt, restart incoming) (duration: 00m 20s) [production]
19:08 <thcipriani@deploy1001> Started deploy [gerrit/gerrit@2de9001]: Gerrit to 2.15.13 (cobalt, restart incoming) [production]
19:06 <thcipriani@deploy1001> Finished deploy [gerrit/gerrit@2de9001]: Gerrit to 2.15.13 (gerrit 2001 only) (duration: 00m 11s) [production]
19:06 <thcipriani@deploy1001> Started deploy [gerrit/gerrit@2de9001]: Gerrit to 2.15.13 (gerrit 2001 only) [production]
18:50 <bblack> repooling cp1085 frontends (weren't meant to be depooled) [production]
18:38 <bblack> re-pooling eqiad front edge traffic (onto new LVSes from T184293 ) [production]
18:35 <XioNoX> update lvs static routes on cr1/2-eqiad - T184293 [production]
18:06 <andrewbogott> restarting rabbitmq-server on cloudcontrol1003 (turning on HA queues) [production]
17:59 <bblack> rebooting lvs1016 in attempt to clear interface config issues - T224027 [production]
17:51 <XioNoX> add BGP sessions to AS202053 in esams [production]
17:31 <bblack> eqiad LVS: low-traffic (all internal services): puppeting lvs1016, bringing back pybal in "secondary" role for all 3 traffic classes (high-traffic1, high-traffic2, low-traffic), no traffic shift expected (again, after merging last-minute fixup https://gerrit.wikimedia.org/r/c/operations/puppet/+/511759 ) [production]
17:25 <bblack> eqiad LVS: low-traffic (all internal services): puppeting lvs1016, bringing back pybal in "secondary" role for all 3 traffic classes (high-traffic1, high-traffic2, low-traffic), no traffic shift expected [production]
17:24 <bblack> eqiad LVS: low-traffic (all internal services): puppeting lvs1006, basically no-op [production]
17:21 <bblack> eqiad LVS: low-traffic (all internal services): puppeting lvs1015, bringing back pybal in primary role, shifting traffic to lvs1015 [production]
17:20 <bblack> eqiad LVS: low-traffic (all internal services): disable pybal on lvs1016 + lvs1015, shifting traffic to lvs1006 [production]
17:18 <reedy@deploy1001> Synchronized php-1.34.0-wmf.6/extensions/Collection/includes/CollectionHooks.php: Fix paths (duration: 00m 56s) [production]
17:17 <bblack> eqiad LVS: high-traffic2 (upload): puppeting lvs1005, basically no-op [production]
17:15 <bblack> eqiad LVS: high-traffic2 (upload): puppeting lvs1002, bringing back pybal in backup role, no traffic shift [production]
17:13 <bblack> eqiad LVS: high-traffic2 (upload): puppeting lvs1014, bringing back pybal in primary role, shifting traffic to lvs1014 [production]
17:11 <bblack> eqiad LVS: high-traffic2 (upload): disable pybal on lvs1014 + lvs1002, shifting traffic to lvs1005 [production]
17:09 <bblack> eqiad LVS: high-traffic1 (text): puppeting lvs1004, basically no-op [production]
17:07 <bblack> eqiad LVS: high-traffic1 (text): puppeting lvs1001, bringing back pybal in backup role, no traffic shift [production]
17:06 <bblack> eqiad LVS: high-traffic1 (text): puppeting lvs1013, bringing back pybal in primary role, shifting traffic to lvs1013 [production]
17:04 <bblack> eqiad LVS: high-traffic1 (text): disable pybal on lvs1013 + lvs1001, shifting traffic to lvs1004 [production]
16:55 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:55 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:55 <jbond42> rebooting wtp1046-1048 [production]
16:55 <bblack> starting Eqiad LVS re-arrangement shortly - T184293 - https://gerrit.wikimedia.org/r/c/operations/puppet/+/511717 (eqiad front edge is still depooled from public traffic) [production]
16:50 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:50 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:50 <jbond42> rebooting wtp1043-1045 [production]
16:46 <mutante> rebooting phab1003 (non-prod) [production]
16:44 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:44 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:44 <jbond42> rebooting wtp1040-1042 [production]
16:39 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:39 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:39 <jbond42> rebooting wtp1037-1039 [production]
16:26 <mobrovac> truncate "others_T_parsoid".data [production]
16:25 <mobrovac> restbase truncate "commons_T_parsoid".data [production]
16:24 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:24 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]