7351-7400 of 10000 results (32ms)
2020-09-28 §
14:40 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=aqs1006.eqiad.wmnet [production]
13:46 <elukey@puppetmaster1001> conftool action : set/pooled=no; selector: name=aqs1006.eqiad.wmnet [production]
09:00 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
08:55 <elukey@cumin1001> START - Cookbook sre.hosts.decommission [production]
08:53 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
08:43 <elukey@cumin1001> START - Cookbook sre.hosts.decommission [production]
08:43 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
08:43 <elukey@cumin1001> START - Cookbook sre.hosts.decommission [production]
08:42 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
08:37 <elukey> decommission the hadoop test cluster (analytics1028->41) [production]
08:36 <elukey@cumin1001> START - Cookbook sre.hosts.decommission [production]
08:36 <elukey@cumin1001> END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97) [production]
08:35 <elukey@cumin1001> START - Cookbook sre.hosts.decommission [production]
08:34 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
08:34 <elukey@cumin1001> START - Cookbook sre.hosts.decommission [production]
08:02 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.stop-cluster (exit_code=0) [production]
07:54 <elukey@cumin1001> START - Cookbook sre.hadoop.stop-cluster [production]
07:17 <elukey@cumin1001> END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) [production]
07:09 <elukey@cumin1001> START - Cookbook sre.presto.roll-restart-workers [production]
2020-09-27 §
06:36 <elukey> powercycle analytics1048 [production]
2020-09-25 §
14:54 <elukey> install linux-image-4.19-amd64 on an-worker1096 + reboot [production]
06:50 <elukey> shutdown ganeti5002 (mistakenly powercycled it without seeing T261130) [production]
06:40 <elukey> powercycle ganeti5002 (no instances running on it, mgmt console shows no tty usable) [production]
06:34 <elukey> reboot stat1004 to pick up kernel settings [production]
2020-09-24 §
13:22 <elukey> moved the hadoop cluster to puppet TLS certificates - T253957 [production]
06:22 <elukey> powercycle elastic2037 (host stuck, no mgmt serial console working, DIMM errors in racadm getsel) [production]
2020-09-23 §
15:52 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) [production]
15:48 <elukey@cumin1001> START - Cookbook sre.hadoop.init-hadoop-workers [production]
15:48 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) [production]
15:45 <elukey@cumin1001> START - Cookbook sre.hadoop.init-hadoop-workers [production]
15:44 <elukey@cumin1001> END (FAIL) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=99) [production]
15:44 <elukey@cumin1001> START - Cookbook sre.hadoop.init-hadoop-workers [production]
15:43 <elukey@cumin1001> END (FAIL) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=99) [production]
15:43 <elukey@cumin1001> START - Cookbook sre.hadoop.init-hadoop-workers [production]
15:40 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) [production]
15:37 <elukey@cumin1001> START - Cookbook sre.hadoop.init-hadoop-workers [production]
15:33 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) [production]
15:30 <elukey@cumin1001> START - Cookbook sre.hadoop.init-hadoop-workers [production]
15:21 <elukey@cumin1001> END (FAIL) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=99) [production]
15:21 <elukey@cumin1001> START - Cookbook sre.hadoop.init-hadoop-workers [production]
09:59 <elukey> update puppet compilter's facts [production]
2020-09-22 §
17:26 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
17:24 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
2020-09-18 §
08:54 <elukey> change analytics-in4/in6 filters on cr1/cr2 after https://gerrit.wikimedia.org/r/628300 [production]
2020-09-16 §
16:09 <elukey> reinstall buster on an-tool1009 after a lot of tests (ganeti VM, so it is a manual work) [production]
12:58 <elukey> upload hue_4.7.1-1+deb10u1 to buster-wikimedia [production]
2020-09-15 §
13:18 <elukey@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) [production]
12:05 <elukey> roll restart cassandra on aqs* to pick up openjdk upgrades [production]
12:05 <elukey@cumin1001> START - Cookbook sre.cassandra.roll-restart [production]
09:08 <elukey@cumin1001> END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) [production]