8701-8750 of 10000 results (11ms)
2018-06-13 §
13:00 <elukey@deploy1001> Finished deploy [analytics/aqs/deploy@84fab89]: Update AQS for T190213 (duration: 02m 38s) [production]
12:57 <elukey@deploy1001> Started deploy [analytics/aqs/deploy@84fab89]: Update AQS for T190213 [production]
12:46 <elukey> restart mirror maker on kafka1012->1014 to pick up new openjdk-7 upgrades [production]
12:28 <elukey> rolling restart of kafka on kafka1012->23 for openjdk-7 upgrades [production]
2018-06-11 §
06:39 <elukey> restart pdfrender on scb1002 [production]
2018-06-08 §
06:44 <elukey> bounce kafka mirror maker main-eqiad-to-main-codfw (kafka200*) due to errors in the logs (also lag metrics not displaying) [production]
2018-06-05 §
11:30 <elukey> manually set net.netfilter.nf_conntrack_tcp_timeout_time_wait to 65 (was 120) on mw* hosts [production]
11:07 <elukey> manually set net.netfilter.nf_conntrack_tcp_timeout_time_wait to 65 (was 120) [production]
2018-06-01 §
14:30 <elukey> killed pt-heartbear-wikimedia after https://gerrit.wikimedia.org/r/436748 on db1107 [production]
2018-05-31 §
10:45 <elukey> removed Pivot from thorium (pivot.wikimedia.org now simply redirects to Turnilo) [production]
09:28 <elukey> reimage druid1006 to Debian Stretch [production]
05:59 <elukey> reimage druid1005 to Debian Stretch [production]
05:51 <elukey> delete /tmp/scap_l10n_1501525840,scap_l10n_1501525840,l10nstuff,l10nstuff3 from tin to free some space in the root partition (1.9G left) [production]
05:40 <elukey> restart pdfrender on scb1001 [production]
2018-05-30 §
13:32 <elukey> reboot analytics1002 (Hadoop master node standby) to pick up new cpu microcode [production]
12:35 <elukey> reboot analytics1029,1042,1070 to pick up the new cpu-microcode [production]
08:47 <elukey> reimage druid1004 to Debian Stretch [production]
06:17 <elukey> reimage druid1001 to Debian stretch [production]
05:50 <elukey> restart Kafka mirror maker on kafka10[12-23] - failures to consume after rebalance [production]
2018-05-29 §
16:59 <elukey> roll restart of kafka mirror maker on kafka-jumbo100* to pick up the new zookeeper settings [production]
16:44 <elukey> roll restart of kafka mirror maker on kafka100[1-3] to pick up new zk settings [production]
15:48 <elukey> roll restart kafka on kafka-jumbo* to pick up new zookeeper settings [production]
15:26 <elukey> restart hadoop yarn/hdfs daemons to pick up the new zookeeper settings [production]
14:24 <elukey> roll restart kafka on kafka100[1-3] (job queues) to pick up the new zookeeper settings [production]
14:03 <elukey> swap zookeeper from conf1003 to conf1006 [production]
07:49 <elukey> reimage druid1002 to debian stretch [production]
06:52 <elukey> roll restart hadoop master daemons to pick up the new zookeeper settings [production]
2018-05-28 §
20:02 <elukey> restart kafka on kafka1003 as attempt to solve the under-replicated partitions warning [production]
19:12 <elukey> roll restart of kafka-mirror maker (main eqiad -> jumbo) on kafka-jumbo* for zookeeper conf updates [production]
18:16 <elukey> restart kafka mirror maker on kafka1012->14 - failed after the last round of kafka restarts [production]
17:26 <elukey> roll restart of kafka on kafka-jumbo* to pick up the new zookeeper settings [production]
17:19 <elukey> restart kafka on kafka1012->23 to pick up the new zookeeper settings [production]
16:31 <elukey> roll restart kafka on kafka100[1-3] to pick up new zookeeper settings [production]
16:21 <elukey> zookeeper cluster restart completed (main-eqiad / conf1*) [production]
16:18 <elukey> stop and mask zookeeper on conf1002 [production]
16:16 <elukey> restart prometheus-burrow-exporter on kafkamon* [production]
15:59 <elukey> swap zookeeper from conf1002 to conf1005 [production]
06:36 <elukey> reimage druid1003 to Debian Stretch (Analytics cluster, backend for Pivot/Turnilo) [production]
2018-05-23 §
12:06 <elukey> upgrade druid public to druid 0.11 (druid100[4-6]) [production]
07:29 <elukey> upload druid debs 0.11.0-3 to stretch-wikimedia [production]
06:44 <elukey> restart zookeeper on druid100[4-6] for openjdk-8 upgrades [production]
2018-05-22 §
16:52 <elukey> restart zookeeper on druid100[1,3] to complete the openjdk-8 upgrade [production]
16:43 <elukey> upload druid debs 0.11.0-3 to jessie-wikimedia [production]
14:07 <elukey> upgrading druid on druid100[123] to 0.11 [production]
13:39 <elukey> upload druid 0.11 debs to jessie|stretch wikimedia [production]
2018-05-17 §
07:25 <elukey> bounced all the prometheus burrow exporters on kafkamon* hosts to refresh their metrics and drop old/expired cgroups [production]
2018-05-16 §
16:22 <elukey> upgrade burrow on kafkamon1001 from 1.0 to 1.1 [production]
15:17 <elukey> upgrade burrow from 1.0.0 to 1.1.0 on kafkamon* hosts [production]
15:17 <elukey> upload burrow 1.1.0 to stretch|jessie-wikimedia [production]
06:19 <elukey> update analytics-in4 on cr1/cr2 eqiad to allow conf100[4-6] (new zookeeper hosts) [production]