7751-7800 of 10000 results (32ms)
2020-06-05 §
10:11 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
09:46 <elukey@cumin1001> START - Cookbook sre.ganeti.makevm [production]
09:25 <elukey@cumin1001> START - Cookbook sre.cassandra.roll-restart [production]
06:20 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
06:17 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
2020-06-04 §
15:12 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=druid1004.eqiad.wmnet [production]
11:49 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:46 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:14 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:12 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:53 <elukey@puppetmaster1001> conftool action : set/pooled=no; selector: name=druid1004.eqiad.wmnet [production]
05:31 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
05:28 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
2020-06-03 §
17:08 <elukey> ganeti: gnd-instance reboot an-launcher1001 to get new memory settings - T254125 [production]
09:08 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:05 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
2020-06-01 §
14:53 <elukey> ganeti: increase memory available for an-launcher1001 from 8g to 12g - T254125 [production]
2020-05-29 §
12:37 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
12:35 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
2020-05-27 §
07:04 <elukey> matomo upgraded to 3.13.5 on matomo1001 - T252741 [production]
06:57 <elukey> update matomo on stretch-wikimedia to 3.13.5 [production]
06:10 <elukey@deploy1001> Finished deploy [analytics/superset/deploy@369a2dd]: Upgrade Superset to 0.36 - second attempt (duration: 00m 57s) [production]
06:09 <elukey@deploy1001> Started deploy [analytics/superset/deploy@369a2dd]: Upgrade Superset to 0.36 - second attempt [production]
2020-05-23 §
08:04 <elukey> powercycle an-presto1004 - unresponsive, racadm getsel shows CPU overheating alerts [production]
2020-05-22 §
09:09 <elukey@deploy1001> Finished deploy [analytics/superset/deploy@be203c8]: Rollback superset to 0.35.2 (duration: 00m 43s) [production]
09:09 <elukey@deploy1001> Started deploy [analytics/superset/deploy@be203c8]: Rollback superset to 0.35.2 [production]
08:18 <elukey@deploy1001> Finished deploy [analytics/superset/deploy@59ba01d]: Upgrade Superset to 0.36 (duration: 01m 01s) [production]
08:17 <elukey@deploy1001> Started deploy [analytics/superset/deploy@59ba01d]: Upgrade Superset to 0.36 [production]
07:07 <elukey@puppetmaster1001> conftool action : set/pooled=yes:weight=10; selector: name=druid1008.eqiad.wmnet [production]
07:04 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=druid1007.eqiad.wmnet [production]
07:04 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=druid1007.eqiad.wmnet [production]
2020-05-21 §
12:29 <elukey> roll restart druid-public cluster (druid100[4-6], backend for the AQS API) to apply new settings + openjdk upgrade - T252771 [production]
2020-05-20 §
15:42 <elukey> update puppet compiler's facts [production]
2020-05-19 §
13:09 <elukey@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
13:09 <elukey@cumin1001> START - Cookbook sre.ganeti.makevm [production]
06:35 <elukey@cumin1001> END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) [production]
06:29 <elukey@cumin1001> START - Cookbook sre.zookeeper.roll-restart-zookeeper [production]
06:24 <elukey@cumin1001> END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) [production]
06:17 <elukey@cumin1001> START - Cookbook sre.zookeeper.roll-restart-zookeeper [production]
2020-05-18 §
15:22 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) [production]
14:02 <elukey@cumin1001> START - Cookbook sre.hadoop.roll-restart-workers [production]
14:00 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) [production]
13:29 <elukey@cumin1001> START - Cookbook sre.hadoop.roll-restart-workers [production]
10:37 <elukey> copy prometheus-druid-exporter 0.8-1 from stretch to buster wikimedia [production]
10:07 <elukey> upload druid 0.12.3-1.1 to stretch|buster-wikimedia [production]
2020-05-15 §
09:09 <elukey> restart druid brokers on druid100[4-6] - locked up due to datasources dropped - T226035 [production]
2020-05-14 §
12:46 <elukey@cumin1001> END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) [production]
12:43 <elukey@cumin1001> START - Cookbook sre.aqs.roll-restart [production]
09:58 <elukey> remove matomo 3.11 from the main component of stretch-wikimedia [production]
09:56 <elukey> upgrade matomo on matomo1001 to 3.13.3 (latest upstream) - T252741 [production]