7251-7300 of 10000 results (24ms)
2020-11-03 §
15:59 <elukey> shutdown kafka-jumbo1006 to replace 1G with 10G nic [production]
10:16 <elukey@deploy1001> Finished deploy [analytics/refinery@cf5db74] (hadoop-test): (no justification provided) (duration: 02m 15s) [production]
10:14 <elukey@deploy1001> Started deploy [analytics/refinery@cf5db74] (hadoop-test): (no justification provided) [production]
10:13 <elukey@deploy1001> Finished deploy [analytics/refinery@cf5db74] (hadoop-test): (no justification provided) (duration: 01m 45s) [production]
10:11 <elukey@deploy1001> Started deploy [analytics/refinery@cf5db74] (hadoop-test): (no justification provided) [production]
2020-11-02 §
13:46 <elukey@cumin1001> END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) [production]
13:40 <elukey> roll restart zookeeper ok an-conf* to pick up new openjdk upgrades [production]
13:40 <elukey@cumin1001> START - Cookbook sre.zookeeper.roll-restart-zookeeper [production]
2020-10-30 §
16:19 <elukey> kafka-jumbo1006 still running with 1g nick [production]
14:42 <elukey> stop kafka-jumbo1006 to swap NICs (1g -> 10g, d1 -> d4 rack) [production]
09:13 <elukey@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
09:07 <elukey@cumin1001> START - Cookbook sre.dns.netbox [production]
08:58 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
08:54 <elukey> decom an-tool1006 (old analytics test vm) - T255139 [production]
08:53 <elukey@cumin1001> START - Cookbook sre.hosts.decommission [production]
2020-10-29 §
14:24 <elukey> restart zookeeper on an-conf1001 for openjdk upgrades [production]
13:54 <elukey> roll out profile::java on all zookeeper instances [production]
10:15 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
10:12 <elukey> restart tilerator on maps100[1,4] - redis errors in the logs [production]
10:11 <elukey> restart tilerator on maps1002 - redis errors in the logs [production]
10:03 <elukey@cumin1001> START - Cookbook sre.ganeti.makevm [production]
10:03 <elukey> drop 10.64.21.6/24 and 2620:0:861:105:10:64:21:6/64 from netbox (an-tool-ui1001 related records) [production]
09:52 <elukey> add gdnsd.service to all gdnsd hosts (with LimitNOFILE=infinity as override) - no daemon restart done - T266746 [production]
09:33 <elukey> clean up 10.64.21.7/24 and 2620:0:861:105:10:64:21:7/64 from netbox (an-test-ui1001 already have ips previously allocated by makevm) [production]
09:32 <elukey@cumin1001> END (ERROR) - Cookbook sre.ganeti.makevm (exit_code=97) [production]
09:23 <elukey@cumin1001> START - Cookbook sre.ganeti.makevm [production]
07:27 <elukey> "sudo truncate -s 10g /var/log/daemon.log" on authdns2001 [production]
06:25 <elukey> execute 'truncate -s 10g /var/log/syslog.1 on authdns2001 - root partition full [production]
2020-10-28 §
18:02 <elukey@cumin1001> END (ERROR) - Cookbook sre.dns.netbox (exit_code=97) [production]
17:46 <elukey@cumin1001> START - Cookbook sre.dns.netbox [production]
15:49 <elukey@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
14:50 <elukey@cumin1001> START - Cookbook sre.ganeti.makevm [production]
10:38 <elukey> clean up 10.64.5.7 and 2620:0:861:104:10:64:5:7 from Netbox (records mistakely allocated via the makevm cookbook) - T266648 [production]
10:35 <elukey@cumin1001> END (ERROR) - Cookbook sre.ganeti.makevm (exit_code=97) [production]
10:20 <elukey@cumin1001> START - Cookbook sre.ganeti.makevm [production]
09:54 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:52 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:50 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:49 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:04 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:02 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
2020-10-27 §
13:15 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) [production]
13:10 <elukey@cumin1001> START - Cookbook sre.hadoop.init-hadoop-workers [production]
13:07 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) [production]
13:04 <elukey@cumin1001> START - Cookbook sre.hadoop.init-hadoop-workers [production]
13:01 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) [production]
12:55 <elukey@cumin1001> START - Cookbook sre.hadoop.init-hadoop-workers [production]
12:55 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) [production]
12:51 <elukey@cumin1001> START - Cookbook sre.hadoop.init-hadoop-workers [production]
11:35 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) [production]