8601-8650 of 10000 results (31ms)
2018-10-05 §
09:37 <elukey> restart rsyslog on lithium - broken connection to tegmen - T199406 [production]
09:06 <elukey> stop etcdmirror replication on conf2002 [production]
2018-10-04 §
17:34 <elukey> pool kafka1002 (eventbus) after maintenance [production]
17:22 <elukey> re-enable ircecho after alarms shower [production]
17:06 <elukey> stop ircecho on einstenium - alarms shower [production]
15:41 <elukey> depool kafka1002 from eventbus as precautionary step for T201039 [production]
12:23 <elukey> deploy etcdmirror on conf1005 - T205814 [production]
07:31 <elukey> move Piwik/Matomo from bohrium to matomo1001 - T202962 [production]
2018-09-26 §
13:38 <elukey> reboot an-master1001 to clear out an issue with systemd@fsck (Hadoop master, failover to an-master1002 included) [production]
08:22 <elukey> start eventlogging_sync on db1108 and the mysql kafka consumers on eventlog1002 after db1107 maintenance [production]
08:04 <elukey> stop eventlogging_sync on db1108 and the mysql kafka consumers on eventlog1002 as prep step for db1107 maintenance [production]
07:47 <elukey> reboot an-master1002 as attempt to clear out some systemd@fsck alarms [production]
2018-09-25 §
12:01 <elukey> end of the maintenance to swap Hadoop masters from analytics100[1,2] to an-master100[1,2] - T203635 [production]
08:03 <elukey> start of the maintenance to swap Hadoop masters from analytics100[1,2] to an-master100[1,2] - T203635 [production]
2018-09-24 §
11:39 <elukey> reboot an-master100[1,2] as part of the pre-checks before the hadoop master daemons swap - T203635 [production]
2018-09-18 §
15:55 <elukey@deploy1001> Finished deploy [analytics/refinery@1a6235a]: Fix cron scrips from the NYC offsite (duration: 09m 32s) [production]
15:46 <elukey@deploy1001> Started deploy [analytics/refinery@1a6235a]: Fix cron scrips from the NYC offsite [production]
2018-09-14 §
08:27 <elukey> reboot kafka100[1-3] (eventbus eqiad) for kernel upgrades [production]
2018-09-13 §
07:47 <elukey> reboot kafka10[12-23] (old analytics cluster) for kernel upgrades [production]
07:46 <elukey> execute apt-get autoremove on notebook* to remove old nginx packages (not used anymore) [production]
06:56 <elukey> reboot notebook100[3,4] for kernel upgrades - T203165 [production]
06:41 <elukey> reboot stat100[4-6] for kernel upgrades - T203165 [production]
2018-09-12 §
12:07 <elukey> delete meitnerium.wikimedia.org's ganeti VM (decommissioned) - T203087 [production]
2018-09-10 §
17:03 <elukey> stop ircecho to avoid excessive spamming [production]
16:37 <elukey> rolling restart of aqs on aqs* to pick new druid backend settings [production]
15:07 <elukey> reboot analytics100[1,2] for kernel security upgrades [production]
14:32 <elukey> reboot analytics1003 for kernel + openjdk-8 upgrades [production]
2018-09-07 §
12:35 <elukey> reboot kafka200[2,3] (eventbus codfw) for kernel + openjdk-8 upgrades [production]
2018-09-06 §
14:38 <elukey> reboot kafka2001 (eventbus codfw host) for kernel + openjdk-8 upgrades [production]
13:45 <elukey> reboot kafka100[2-6] for kernel + openjdk-8 upgrades [production]
13:24 <elukey> reboot kafka-jumbo1001 for openjdk-8 + kernel security upgrades [production]
2018-09-05 §
14:50 <elukey@deploy1001> Finished deploy [analytics/refinery@77a5a83]: small changes to pageview whitelist and scripts (duration: 08m 40s) [production]
14:42 <elukey@deploy1001> Started deploy [analytics/refinery@77a5a83]: small changes to pageview whitelist and scripts [production]
14:07 <elukey> reboot druid* hosts for kernel + openjdk-8 upgrades [production]
09:07 <elukey> rebooting druid1001 for kernel + openjdk-8 upgrades [production]
08:22 <elukey> reboot aqs100[5-9] for kernel + openjdk-8 upgrades [production]
07:48 <elukey> re-enable puppet on mc1035 - memcache unit refreshed, mw cache shard wiped - T203429 [production]
07:02 <elukey> restart oozie on analytics1003 to pick up new smtp settings [production]
2018-09-04 §
16:31 <elukey> reboot aqs1004 again for kernel + openjdk-8 upgrades (now available since the root partition is not read only anymore) [production]
15:30 <elukey> reboot aqs1004 after running fsck on root (was read-only) [production]
13:06 <elukey> restart memcached on mc1035 with -v option - T203429 [production]
10:17 <elukey> restart ircecho on einstenium to force it re-join #wikimedia-analytics [production]
09:32 <elukey> restart ircecho as attempt to see if icinga-vm re-joins the analytics chan [production]
2018-08-31 §
05:54 <elukey> resumed the Hadoop workers reboots for kernel upgrades [production]
05:33 <elukey> restart pdfrender on scb1003 [production]
2018-08-30 §
12:59 <elukey> drain + reboot analytics10[28-79]* for kernel updates (will take multiple days) [production]
2018-08-29 §
15:48 <elukey> roll restart aqs on aqs100[4-9] to pick up the new druid config settings [production]
10:41 <elukey@deploy1001> Finished deploy [analytics/refinery@1c6423f]: (no justification provided) (duration: 02m 56s) [production]
10:38 <elukey@deploy1001> Started deploy [analytics/refinery@1c6423f]: (no justification provided) [production]
09:33 <elukey> cr1/2-eqiad: update analytics-in4 filter with the new archiva host, add a new term 'archiva' to analytics-in6 filter - T198623 [production]