7101-7150 of 10000 results (39ms)
2020-11-24 §
15:38 <elukey> move druid1005 from rack B7 to B6 - T267065 [production]
14:58 <elukey> move analytics1072 from rack B2 to B3 - T267065 [production]
13:52 <elukey@deploy1001> Finished deploy [statsv/statsv@b25b6ff]: Deploy https://gerrit.wikimedia.org/r/c/analytics/statsv/+/643252 (duration: 00m 05s) [production]
13:52 <elukey@deploy1001> Started deploy [statsv/statsv@b25b6ff]: Deploy https://gerrit.wikimedia.org/r/c/analytics/statsv/+/643252 [production]
09:09 <elukey> drop principals and keytabs for analytics10[42-57] - T267932 [production]
08:48 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
08:42 <elukey@cumin1001> START - Cookbook sre.hosts.decommission [production]
08:42 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
08:36 <elukey@cumin1001> START - Cookbook sre.hosts.decommission [production]
08:28 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) [production]
08:27 <elukey@cumin1001> START - Cookbook sre.hosts.decommission [production]
2020-11-23 §
17:12 <elukey> move aqs1004 from rack A4 to A3 - T267065 [production]
16:37 <elukey> move analytics1070 from rack A7 to rack A5 - T267065 [production]
11:13 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) [production]
11:13 <elukey@cumin1001> START - Cookbook sre.hosts.decommission [production]
08:46 <elukey> drop kerberos keytabs for analytics10[28-41] from krb1001:/srv/kerberos/keytabs, decommed nodes (old hadoop test cluster) [production]
08:41 <elukey> drop kerberos principals from krb1001 for analytics10[29-41], decommed nodes (old hadoop test cluster) [production]
08:36 <elukey> drop analytics1028's krb principals from krb1001 - old decommed node [production]
2020-11-21 §
08:10 <elukey> remove big stderrlog fine in /var/lib/hadoop/data/d/yarn/logs/application_1605880843685_1450 on an-worker1110 [production]
08:05 <elukey> remove big stderrlog fine in /var/lib/hadoop/data/e/yarn/logs/application_1605880843685_1450 on an-worker1105 [production]
2020-11-20 §
18:47 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
18:42 <elukey@cumin1001> START - Cookbook sre.hosts.decommission [production]
18:37 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
18:31 <elukey@cumin1001> START - Cookbook sre.hosts.decommission [production]
18:31 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
18:18 <elukey@cumin1001> START - Cookbook sre.hosts.decommission [production]
14:30 <elukey> force umount/mount for /mnt/hdfs on all stat1* nodes to pick up new openjdk settings [production]
14:28 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.roll-restart-masters (exit_code=0) [production]
14:00 <elukey> restart hadoop daemons on an-master[1001-1002] (Hadoop masters) to pick up new rack settings and openjdk upgrades [production]
13:59 <elukey@cumin1001> START - Cookbook sre.hadoop.roll-restart-masters [production]
12:54 <elukey@cumin1001> END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) [production]
09:56 <elukey> update analytics filters on cr1/cr2 eqiad (ref: https://gerrit.wikimedia.org/r/c/operations/homer/public/+/642346) [production]
08:57 <elukey@cumin1001> START - Cookbook sre.kafka.roll-restart-brokers [production]
08:50 <elukey@cumin1001> END (PASS) - Cookbook sre.kafka.roll-restart-mirror-maker (exit_code=0) [production]
08:32 <elukey@cumin1001> START - Cookbook sre.kafka.roll-restart-mirror-maker [production]
08:31 <elukey> roll restart kafka daemons on kafka-jumbo100* to pick up openjdk upgrades [production]
08:10 <elukey> update analytics filters on cr1/cr2 eqiad (ref: https://gerrit.wikimedia.org/r/642268) [production]
2020-11-19 §
18:03 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) [production]
16:35 <elukey> roll restart hadoop workers for openjdk upgrades [production]
16:35 <elukey@cumin1001> START - Cookbook sre.hadoop.roll-restart-workers [production]
16:06 <elukey@cumin1001> END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) [production]
15:56 <elukey@cumin1001> START - Cookbook sre.presto.roll-restart-workers [production]
15:43 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
15:40 <elukey@cumin1001> START - Cookbook sre.hosts.decommission [production]
15:40 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
15:36 <elukey@cumin1001> START - Cookbook sre.hosts.decommission [production]
15:36 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
15:30 <elukey@cumin1001> START - Cookbook sre.hosts.decommission [production]
15:29 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
15:26 <elukey@cumin1001> START - Cookbook sre.hosts.decommission [production]