7501-7550 of 10000 results (30ms)
2020-09-03 §
16:24 <elukey> roll restart aqs on aqs1* to pick up new druid settings [production]
2020-09-02 §
14:31 <elukey> execute kafka topics --alter --topic codfw.resource-purge --partitions 3 and kafka topics --alter --topic eqiad.resource-purge --partitions 3 on kafka-main eqiad - T261865 [production]
14:28 <elukey> execute kafka topics --alter --topic codfw.resource-purge --partitions 3 and kafka topics --alter --topic eqiad.resource-purge --partitions 3 on kafka-main codfw - T261865 [production]
13:05 <elukey> run kafka preferred-replica-election on kafka-main codfw [production]
09:19 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:16 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:01 <elukey> reimage kafka-jumbo1004 to Buster [production]
07:58 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
07:56 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
07:38 <elukey> reimage kafka-jumbo1003 to buster [production]
07:02 <elukey> reboot kafka-jumbo1002 to pick up new kernel settings [production]
06:38 <elukey> powercycle analytics1059 - cpu soft locks on multiple CPUs [production]
06:30 <elukey> reboot kafka-jumbo1001 to pick up new kernel settings [production]
2020-09-01 §
06:59 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
06:57 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
06:44 <elukey> reimage kafka-jumbo1002 to Buster [production]
2020-08-31 §
13:05 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:02 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:58 <elukey> reimage kafka-jumbo1001 to Buster [production]
09:51 <elukey> executed /srv/phab/phabricator/bin/remove destroy @klausman on phab1001 (following https://wikitech.wikimedia.org/wiki/Phabricator#Delete_a_user) to clear incosistent state of new account (wrong email address) [production]
06:30 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
06:27 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
06:06 <elukey> reimage kafka-jumbo1005 to Debian Buster [production]
2020-08-28 §
13:25 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:23 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:07 <elukey> stop kafka on kafka-jumbo1006 and reimage to buster [production]
2020-08-27 §
16:12 <elukey> remove some old/stale terms from analytics-in4 on cr1/cr2-eqiad (ref: https://gerrit.wikimedia.org/r/c/operations/homer/public/+/622746, https://gerrit.wikimedia.org/r/c/operations/homer/public/+/622744) [production]
14:42 <elukey> add eventgate-related terms to analytics-in4 filter on cr1/cr2-eqiad (ref https://gerrit.wikimedia.org/r/c/operations/homer/public/+/622705) [production]
13:29 <elukey> restart jvm daemons on analytics1042, aqs1004, kafka-jumbo1001 to pick up new openjdk upgrades (canaries) [production]
09:45 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
09:44 <elukey@cumin1001> START - Cookbook sre.hosts.decommission [production]
09:44 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
09:43 <elukey@cumin1001> START - Cookbook sre.hosts.decommission [production]
09:43 <elukey> decommissioning vms schema[12]00[12] (replaced previously by schema[12]00[34] buster vms) [production]
09:42 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
09:41 <elukey@cumin1001> START - Cookbook sre.hosts.decommission [production]
09:40 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
09:39 <elukey@cumin1001> START - Cookbook sre.hosts.decommission [production]
06:50 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
06:49 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
06:46 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
06:45 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
06:34 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
06:31 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
2020-08-26 §
14:45 <elukey@puppetmaster1001> conftool action : set/pooled=inactive:weight=0; selector: name=schema1002.eqiad.wmnet [production]
14:45 <elukey@puppetmaster1001> conftool action : set/pooled=inactive:weight=0; selector: name=schema1001.eqiad.wmnet [production]
14:45 <elukey@puppetmaster1001> conftool action : set/pooled=inactive:weight=0; selector: name=schema2001.codfw.wmnet [production]
14:44 <elukey@puppetmaster1001> conftool action : set/pooled=inactive:weight=0; selector: name=schema2002.codfw.wmnet [production]
14:34 <elukey@puppetmaster1001> conftool action : set/pooled=yes:weight=10; selector: name=schema2003.codfw.wmnet [production]
14:34 <elukey@puppetmaster1001> conftool action : set/pooled=yes:weight=10; selector: name=schema2004.codfw.wmnet [production]