9351-9400 of 10000 results (31ms)
2017-08-07 §
12:39 <elukey> restart kafka on kafka1018 to force it out of the kafka topic leaders - T172681 [production]
09:06 <elukey> set net.netfilter.nf_conntrack_tcp_timeout_time_wait=65 (was 120) on all the analytics kafka brokers - T136094 [production]
2017-08-06 §
13:17 <elukey> powercycle mw2256 - com2 frozen - T163346 [production]
13:13 <elukey> restart pdfrender on scb1002 [production]
2017-08-03 §
13:28 <elukey> drop CookieBlock backup tables for T171883 [production]
10:21 <elukey> removed /run/pacct_shadow.d on stat1005 to allow atopacct.service restart [production]
2017-08-01 §
12:49 <elukey> restart hive daemons on analytics1003 to pick up new jvm settings (bigger Xmx, JMX ports) [production]
12:04 <elukey> stop eventlogging_sync on analytics-slaves && rename all CookieBlock* tables (log db) to CookieBlock*_backup - T171883 [production]
2017-07-31 §
08:55 <elukey> update nodejs* on aqs100[56789] to 6.11 - T170790 [production]
2017-07-28 §
09:41 <elukey> re-enable irc-echo on einstenium [production]
08:41 <elukey> stop ircecho on einstenium as puppet-error-shower countermeasure [production]
07:56 <elukey> update nodejs to 6.11 on aqs1004 (testing prod node after beta qa) - T170790 [production]
07:52 <elukey> forced mii-tool -r eth0 on analytics1034 to get 1G negotiated speed [production]
2017-07-26 §
08:47 <elukey> rollout logster 0.0.10-2~jessie1 to the cache hosts [production]
08:46 <elukey> upload logster 0.0.10-2~jessie1 to jessie-wikimedia [production]
2017-07-25 §
12:56 <elukey> rolling restart of aqs* for jvm upgrades [production]
2017-07-20 §
07:41 <elukey> powercycle cp3048 - mgmt reachable - T171145 [production]
2017-07-19 §
13:45 <elukey> restart hive-server on analytics1003 - Java OOM issue due to a huge query [production]
13:32 <elukey> Limit the access to the conf* zookeeper ports via ferm rules - https://gerrit.wikimedia.org/r/366228 [production]
2017-07-18 §
09:08 <elukey> reboot conf1003 for kernel updates [production]
09:00 <elukey> reboot conf1002 for kernel updates [production]
07:32 <elukey> moved /home to /srv/home on stat1006 to free disk space (created symling from /home -> /srv/home too) - T152712 [production]
2017-07-17 §
14:46 <elukey> reboot conf1001 for kernel updates [production]
2017-07-15 §
10:42 <elukey> puppetdb restarted on nitrogen, puppet agents re-enabled - T170740 [production]
2017-07-11 §
15:24 <elukey> restart burrow on krypton [production]
2017-07-10 §
15:56 <elukey@tin> Finished deploy [analytics/refinery@6da2774]: Update stat1002 with the last refinery deployment (duration: 00m 04s) [production]
15:55 <elukey@tin> Started deploy [analytics/refinery@6da2774]: Update stat1002 with the last refinery deployment [production]
2017-07-07 §
12:20 <elukey> restart mysql on dbstore1002 - high swap used [production]
2017-07-06 §
11:15 <elukey> reboot conf2003 for kernel updates [production]
10:52 <elukey> reboot conf2002 for kernel update [production]
2017-07-05 §
13:16 <elukey> reboot conf2001 for kernel updates [production]
2017-07-04 §
11:15 <elukey> powercycle elastic1018, host unreachable [production]
08:25 <elukey> restart redis 6380 (slave) jobqueue instance on rdb1004/2003 to force resync with master [production]
2017-07-03 §
14:40 <elukey> running EventLogging alter tables on dbstore1002 (script in /home/elukey/dbstore1002.sql) - T167162 [production]
12:26 <elukey> reimage stat1005 with Debian Stretch [production]
2017-06-30 §
10:55 <elukey> deploy kafkatee 0.1.6-1 to oxygen [production]
09:54 <elukey> uploaded kafkatee 0.1.6-1 to reprepro - T151748 [production]
2017-06-29 §
10:57 <elukey@tin> Finished deploy [analytics/refinery@f6cccf9]: Weekely refinery deployment (duration: 02m 56s) [production]
10:54 <elukey@tin> Started deploy [analytics/refinery@f6cccf9]: Weekely refinery deployment [production]
10:53 <elukey@tin> Finished deploy [analytics/refinery@f6cccf9]: Weekely refinery deployment (duration: 00m 11s) [production]
10:53 <elukey@tin> Started deploy [analytics/refinery@f6cccf9]: Weekely refinery deployment [production]
08:19 <elukey> restart pdfrender on scb1004 - xpra issue [production]
07:36 <elukey> depooled kafka2001.codfw.wmnet for T168462 [production]
07:19 <elukey@tin> Finished deploy [analytics/refinery@f6cccf9]: Updated stat1002 with the last refinery deployment (duration: 02m 55s) [production]
07:16 <elukey@tin> Started deploy [analytics/refinery@f6cccf9]: Updated stat1002 with the last refinery deployment [production]
07:12 <elukey@tin> Finished deploy [analytics/refinery@f6cccf9]: Updated stat1002 with the last refinery deployment (duration: 02m 36s) [production]
07:10 <elukey@tin> Started deploy [analytics/refinery@f6cccf9]: Updated stat1002 with the last refinery deployment [production]
2017-06-28 §
11:14 <elukey> stop eventlogging_sync on db1047 - alter tables running [production]
07:01 <elukey> stop jobrunner/jobchron on mw130[4,5,6] and reboot them for kernel updates [production]
06:42 <elukey> stop jobrunner/jobchron on mw130[2,3] and reboot them for kernel updates [production]