9001-9050 of 10000 results (30ms)
2018-02-23 §
14:59 <elukey> update facts on puppet compiler [production]
10:01 <elukey> restart hhvm on mw1230 [production]
09:54 <elukey> restart hhvm on mw1286 [production]
09:50 <elukey> restart hhvm on mw1227 [production]
2018-02-22 §
17:23 <elukey> installed linux-perf-4.9 on phab1001 to experiment with perf tracing [production]
15:24 <elukey> manually removing from cp1008 and cache::misc old files related to the varnishkafka jumbo testing instance (after https://gerrit.wikimedia.org/r/413370) [production]
2018-02-21 §
21:50 <elukey> restart hhvm on mw1224 - high load alarms [production]
21:46 <elukey> restart hhvm on mw1235 - high load alarms [production]
21:44 <elukey> restart hhvm on mw1233 - high load alarms [production]
21:34 <elukey> restart hhvm on mw1232 - high load alarms [production]
21:30 <elukey> restart hhvm on mw1229 - high load alarms [production]
21:27 <elukey> restart hhvm on mw1227 - high load alarms [production]
21:23 <elukey> restart hhvm on mw1221 - high load alarms [production]
12:38 <elukey> restart hhvm on mw1234 - high load [production]
12:26 <elukey> restart hhvm on mw1231 - high load, hhvm-dump-debug in /home/elukey/hhvm.6759.bt [production]
12:21 <elukey> restart hhvm on mw1227 - high load, hhvm-dump-debug in /home/elukey/hhvm.23382.bt [production]
2018-02-20 §
09:14 <elukey> restart zookeeper on druid1001 (follower) to verify that the last changes are no-op [production]
2018-02-19 §
17:04 <elukey@tin> Finished deploy [eventlogging/analytics@8bebdf7]: (no justification provided) (duration: 00m 05s) [production]
17:04 <elukey@tin> Started deploy [eventlogging/analytics@8bebdf7]: (no justification provided) [production]
2018-02-16 §
11:09 <elukey> restart nfaccd on rhenium to see if it picks up the new kafka topic config (3 partitions) [production]
2018-02-14 §
13:44 <elukey> rollback java 8 upgrade for archiva - issues with Analytics builds [production]
13:34 <elukey> installed openjdk-8 on meitnerium, manually upgraded java-update-alternatives to java8, restarted archiva [production]
2018-02-13 §
18:25 <elukey> Analytics Hadoop cluster upgrade to Java 8 about to start - complete cluster shutdown is needed - T166248 [production]
09:22 <elukey> powercycle analytics1062 - not reachable via ssh, frozen via serial console [production]
2018-02-12 §
23:13 <elukey> manual restart of Yarn Node Managers on analytics1058/31 (failed due to root partition filled up for the issue logged before) [production]
23:09 <elukey> cleaned up tmp files on all analytics hadoop worker nodes, job filling up tmp [production]
17:18 <elukey> home dirs on stat1004 moved to /srv/home (/home symlinks to it) [production]
14:54 <elukey> upload prometheus-burrow-exporter 0.0.4 on jessie/stretch-wikimedia [production]
09:51 <elukey> reboot mw1302 (hhvm defunct processes, hungs registered in dmesg, very high load) [production]
2018-02-09 §
07:39 <elukey> forced remount of /mnt/hdfs on stat1005 [production]
2018-02-08 §
16:23 <elukey> stop archiva on meitnerium to swap /var/lib/archiva from the root partition to a new separate one - T186020 [production]
2018-02-07 §
11:08 <elukey> install libc6-dbg on phab1001 to get a more precise gdb stack trace - T182832 [production]
2018-02-06 §
16:56 <elukey> restart httpd on phab1001 [production]
15:36 <elukey> drain + shutdown of analytics1038 to replace faulty BBU - T185409 [production]
08:21 <elukey> rollback apache/httpd changes on phab1001 (restart required) [production]
2018-02-05 §
19:05 <elukey> executed 'echo '/srv/apache2_dump/core.%h.%e.%p.%t' > /proc/sys/kernel/core_pattern' on phab1001 - T182832 [production]
18:35 <elukey> add 'ulimit -c unlimited' to /etc/default/apache2 to see if httpd's CoreDumpDirectory works properly on phab1001 [production]
15:26 <elukey> temporary setting CoreDumpDirectory /srv/apache2_dump to httpd on phab1001 (+ httpd reload) to investigate core dumps for T182832 [production]
11:03 <elukey> restart eventlogging/forwarder legacy-zmq on eventlog1001 due to slow memory leak over time (cached memory down to zero) [production]
07:43 <elukey> install libjson-c2-dbg on phab1001 to allow better debugging of httpd/mod-php stuck process - T182832 [production]
2018-02-04 §
22:40 <elukey> restart aphlict.service on phab1001 to force it to pick up the new logfile (/var/log/aphlict/aphlict.log rather than the .log.1) [production]
2018-02-02 §
20:47 <elukey> truncated /var/log/aphlict/aphlict.log to 1G (was 26G) to avoid overhead for the upcoming first logrotate [production]
13:59 <elukey> reboot meitnerium via gnt-instance reboot on ganeti1005 to pick up new disk config - T184794 [production]
08:37 <elukey> apt-get install php5-dbg on phab1001 as attempt to have a better gdb output for T182832 [production]
05:37 <elukey> truncate /var/log/aphlict/aphlict.log to 25G as temp measure to avoid phab1001's root partition to fill up [production]
2018-01-31 §
06:19 <elukey> restart varnish backend on cp4024 - failed fetches / 503s [production]
2018-01-23 §
06:50 <elukey> restart varnish backend on cp4021, 503s and mailbox lag [production]
2018-01-22 §
14:36 <elukey> truncate (again) /var/log/upstart/neutron-server.log on labtestnet2001 [production]
07:04 <elukey> truncated /var/log/upstart/neutron-server.log on labtestnet2001 - / disk space exhausted [production]
2018-01-20 §
17:36 <elukey> forced bbu learn cycle on analytics1038 (cache policy flapping from WriteBack to WriteThrough) [production]