8051-8100 of 10000 results (24ms)
2019-11-27 §
07:49 <elukey> roll restart of eventstreams on scb2* - T239220 [production]
2019-11-26 §
11:36 <elukey> reboot stat1007 [production]
2019-11-25 §
14:50 <elukey@cumin1001> END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) [production]
14:44 <elukey@cumin1001> START - Cookbook sre.zookeeper.roll-restart-zookeeper [production]
13:27 <elukey> set global read_only=1 on db1108's log database - T159170 [production]
2019-11-19 §
19:00 <elukey> regenerate TLS cert for yarn.wikimedia.org (containing SANs for all analytics UIs) to add datasets.w.o SAN (site was failing due to ATS not being able to contact thorium) [production]
2019-11-16 §
09:43 <elukey> systemctl restart hadoop-* on analytics1077 after oom killer [production]
2019-11-12 §
07:36 <elukey> remove /etc/logrotate.d/wdqs_autodeployment_log from wdqs1009 (not in puppet anymore and causing cronspam) [production]
2019-11-11 §
17:53 <elukey@cumin1001> END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) [production]
15:44 <elukey@cumin1001> START - Cookbook sre.kafka.roll-restart-brokers [production]
15:42 <elukey@cumin1001> END (PASS) - Cookbook sre.kafka.roll-restart-mirror-maker (exit_code=0) [production]
15:30 <elukey@cumin1001> START - Cookbook sre.kafka.roll-restart-mirror-maker [production]
06:56 <elukey> delete /etc/logrotate.d/wdqs-reload-categories from wdqs* as attempt to reduce cronspam [production]
2019-11-08 §
13:27 <elukey@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) [production]
12:34 <elukey@cumin1001> START - Cookbook sre.cassandra.roll-restart [production]
10:09 <elukey> restart jvm-based hadoop daemons on an-master100[1,2] to pick up the new openjdk version [production]
09:35 <elukey@cumin1001> END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) [production]
09:05 <elukey@cumin1001> START - Cookbook sre.druid.roll-restart-workers [production]
08:57 <elukey@cumin1001> END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) [production]
08:31 <elukey@cumin1001> START - Cookbook sre.druid.roll-restart-workers [production]
08:23 <elukey> restart kafka on kafka-jumbo1001 to test the new openjdk [production]
2019-11-07 §
17:44 <elukey@cumin1001> END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) [production]
17:39 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) [production]
17:38 <elukey@cumin1001> START - Cookbook sre.zookeeper.roll-restart-zookeeper [production]
17:37 <elukey@cumin1001> END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) [production]
17:30 <elukey@cumin1001> START - Cookbook sre.zookeeper.roll-restart-zookeeper [production]
17:09 <elukey@cumin1001> START - Cookbook sre.hadoop.roll-restart-workers [production]
17:00 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) [production]
15:40 <elukey@cumin1001> START - Cookbook sre.hadoop.roll-restart-workers [production]
2019-11-04 §
13:38 <elukey> update bacula terms on analytics-in{4,6} filters on cr{1,2}-eqiad - T237016 [production]
2019-10-31 §
06:37 <elukey> upgrade cergen to 0.2.5 on puppetmaster1001 [production]
2019-10-30 §
17:43 <elukey> upload cergen 0.2.5-1+deb10u1 to buster-wikimedia component/cergen [production]
17:41 <elukey> run reprepro clearvanished on install1002 to clean leftovers of buster-wikimedia|thirdparty/elastic7 [production]
2019-10-29 §
14:29 <elukey> upgrade python-kafka on webperf[12]001 - T234808 [production]
07:06 <elukey> roll restart java daemons on analytics1042, druid1003 and aqs1004 to pick up new openjdk upgrades [production]
2019-10-28 §
17:20 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
16:56 <elukey@cumin1001> START - Cookbook sre.ganeti.makevm [production]
07:40 <elukey@deploy1001> Finished deploy [eventlogging/analytics@0f1ad6d]: Move codebase to Python3 - second attempt (duration: 00m 05s) [production]
07:40 <elukey@deploy1001> Started deploy [eventlogging/analytics@0f1ad6d]: Move codebase to Python3 - second attempt [production]
07:37 <elukey> upload archiva 2.2.4-1 to wikimedia-stretch (fix to avoid overriding archiva.xml upon install) [production]
2019-10-17 §
18:01 <elukey> update librdkafka on eventlog1002 and restart eventlogging [production]
10:25 <elukey> rollback eventlogging back to Python 2, some errors (unseen in tests) logged by the processors [production]
10:24 <elukey@deploy1001> Finished deploy [eventlogging/analytics@0f0a1aa]: Rollback move codebase to Python3 (duration: 00m 03s) [production]
10:24 <elukey@deploy1001> Started deploy [eventlogging/analytics@0f0a1aa]: Rollback move codebase to Python3 [production]
10:19 <elukey> Move eventlogging on eventlog1002 to Python3 [production]
10:17 <elukey@deploy1001> Finished deploy [eventlogging/analytics@0f0a1aa]: Move codebase to Python3 (duration: 00m 05s) [production]
10:17 <elukey@deploy1001> Started deploy [eventlogging/analytics@0f0a1aa]: Move codebase to Python3 [production]
09:27 <elukey> upload archiva 2.2.4-1 to stretch-wikimedia - T222595 [production]
06:06 <elukey> upgrade archiva on archiva1001 to 2.2.4 - T222595 [production]
2019-10-16 §
10:31 <elukey> upload prometheus-memcached-exporter 0.4.1+git20181010.2fa99eb-1+deb10u1 to buster-wikimedia - T213089 [production]