7551-7600 of 10000 results (22ms)
2020-08-26 §
14:33 <elukey@puppetmaster1001> conftool action : set/pooled=yes:weight=10; selector: name=schema1004.eqiad.wmnet [production]
14:33 <elukey@puppetmaster1001> conftool action : set/pooled=yes:weight=10; selector: name=schema1003.eqiad.wmnet [production]
10:30 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:28 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:14 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
10:14 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:18 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:14 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:16 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:14 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
2020-08-25 §
15:47 <elukey> restart mariadb@analytics_meta on db1108 to apply a replication filter (exclude superset_staging database from replication) [production]
2020-08-13 §
14:05 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
14:00 <elukey> create schema[12]00[34] in ganeti - T260347 [production]
13:59 <elukey@cumin1001> START - Cookbook sre.ganeti.makevm [production]
13:58 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
13:53 <elukey@cumin1001> START - Cookbook sre.ganeti.makevm [production]
13:51 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
13:46 <elukey@cumin1001> START - Cookbook sre.ganeti.makevm [production]
13:44 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
13:39 <elukey@cumin1001> START - Cookbook sre.ganeti.makevm [production]
2020-08-11 §
10:20 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.change-distro-from-cdh (exit_code=0) [production]
10:01 <elukey@cumin1001> START - Cookbook sre.hadoop.change-distro-from-cdh [production]
10:00 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.stop-cluster (exit_code=0) [production]
09:51 <elukey@cumin1001> START - Cookbook sre.hadoop.stop-cluster [production]
2020-08-07 §
10:02 <elukey> reboot deneb via ganeti2021 (hostname config pointing to recdns for some reason) [production]
2020-08-06 §
07:03 <elukey@cumin1001> END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) [production]
06:57 <elukey@cumin1001> START - Cookbook sre.zookeeper.roll-restart-zookeeper [production]
06:53 <elukey@cumin1001> END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) [production]
06:47 <elukey@cumin1001> START - Cookbook sre.zookeeper.roll-restart-zookeeper [production]
06:43 <elukey@cumin1001> END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) [production]
06:37 <elukey> roll restart of druid clusters' zookeeper and an-conf* zookeeper for openjdk-11 upgrades [production]
06:36 <elukey@cumin1001> START - Cookbook sre.zookeeper.roll-restart-zookeeper [production]
2020-08-05 §
16:50 <elukey> powercycle stat1005 after GPU issue [production]
14:48 <elukey> reboot stat1008 for unexpected maintenance (GPU stuck) [production]
13:04 <elukey> restart yarn resource managers on an-master100[12] to pick up new Yarn settings - https://gerrit.wikimedia.org/r/c/operations/puppet/+/618529 [production]
09:32 <elukey> set ticket max renewable lifetime to 7d on all kerberos clients (was zero, the default) [production]
2020-08-04 §
07:34 <elukey> upgrade druid analytics (backend for Turnilo/Superset/etc..) to 0.19 [production]
2020-08-03 §
08:07 <elukey> roll restart aqs on aqs* to pick up new druid settings [production]
2020-07-31 §
13:52 <elukey> update cr1/cr2-eqiad's analytics filters (ref: https://gerrit.wikimedia.org/r/c/operations/homer/public/+/617649/) [production]
07:07 <elukey> stop mysql replication on db1108; update port config for mysql instances and restart them; restart replication on instances [production]
06:32 <elukey> roll restart of druid brokers on druid100[4-8] to pick up new changes [production]
2020-07-30 §
12:07 <elukey> upgrade of the druid public cluster (serving AQS) from 0.12.3 to 0.19 [production]
06:57 <elukey> upload druid_0.19.0-1 packages to buster-wikimedia [production]
2020-07-27 §
06:44 <elukey> truncate big log file on an-launcher1002 that is filling up the /srv partition [production]
06:36 <elukey> apt-get clean on netbox1001 to free some space [production]
2020-07-24 §
07:44 <elukey> depool wtp1025 - disk full [production]
2020-07-22 §
06:47 <elukey> update analytics-in4/6 filters on cr1/cr2 eqiad (ref https://gerrit.wikimedia.org/r/c/operations/homer/public/+/614702) [production]
2020-07-21 §
09:44 <elukey> add term 'idp' to analytics-in4/6 filters on cr1-eqiad and cr2-eqiad (ref: https://gerrit.wikimedia.org/r/c/operations/homer/public/+/615160) [production]
2020-07-20 §
15:59 <elukey> restart airflow-webserver/scheduler to pick up TLS to mysql settings [production]
06:55 <elukey> restart matomo1002's mariadb to pick up new TLS settings [production]