2351-2400 of 2992 results (17ms)
2017-01-10 §
19:05 <madhuvishy> Killed 3 jobs from tools.arnaub that were causing high load on tools-exec-1411 [tools]
2017-01-06 §
19:02 <bd808> Terminated deprecated instances tools-exec-121[2-6] (T154539) [tools]
2017-01-04 §
02:43 <madhuvishy> Reenabled puppet on toolschecker and removed iptables rule on labservices1001 blocking incoming connections from tools-checker-01. T152369 [tools]
2017-01-03 §
23:56 <bd808> Removed tools-exec-12[12-16] from gridengine (T154539) [tools]
23:27 <bd808> drained tools-exec-1216 (T154539) [tools]
23:26 <bd808> drained tools-exec-1215 (T154539) [tools]
23:25 <bd808> drained tools-exec-1214 (T154539) [tools]
23:25 <bd808> drained tools-exec-1213 (T154539) [tools]
23:24 <bd808> drained tools-exec-1212 (T154539) [tools]
23:11 <madhuvishy> Disabled puppet on tools-checker-01 (T152369) [tools]
21:43 <madhuvishy> Adding iptables rule to drop incoming connections from toolschecker on labservices1001 [tools]
20:51 <madhuvishy> Adding iptables rule to block outgoing connections to labservices1001 on tools-checker-01 [tools]
20:43 <madhuvishy> Silenced tools checker on icinga to test labservices1001 failure causing toolschecker to flake out T152369 [tools]
2016-12-25 §
00:28 <yuvipanda> comment out cron running 'clean' script of avicbot every minute without -once [tools]
00:28 <yuvipanda> force delete all jobs of avicbot [tools]
00:25 <yuvipanda> delete all jobs of avicbot. This is 419 jobs [tools]
00:20 <yuvipanda> kill clean.sh process of avicbot [tools]
2016-12-19 §
20:07 <valhallasw`cloud> killed gps_exif_bot2.py (tools.gpsexif), was using 50MB/s io, lagging all of tools-bastion-03 [tools]
13:06 <yuvipanda> run /usr/local/bin/deploy-master http://tools-docker-builder-03.tools.eqiad.wmflabs v1.3.3wmf1 on tools-k8s-master-01 [tools]
12:53 <yuvipanda> cleaned out pbuilder from tools-docker-builder-01 to clean up [tools]
2016-12-17 §
04:49 <yuvipanda> turned on lookupcache again for bastions [tools]
2016-12-15 §
18:52 <yuvipanda> reboot tools-exec-1204 [tools]
18:49 <yuvipanda> reboot tools-webgrid-lighttpd-12[01-05] [tools]
18:45 <yuvipanda> reboot tools-exec-gift [tools]
18:41 <yuvipanda> reboot tools-exec-1217 to 1221 [tools]
18:30 <yuvipanda> rebooted tools-exec-1212 to 1216 [tools]
14:55 <yuvipanda> reboot tools-services-01 [tools]
2016-12-14 §
18:43 <mutante> tools-bastion-03 - ran 'locale-gen ko_KR.EUC-KR' for T130532 [tools]
2016-12-13 §
20:54 <chasemp> reboot bastion-03 as unresponsive [tools]
2016-12-09 §
19:32 <godog> upgrade / restart prometheus-node-exporter [tools]
08:37 <YuviPanda> run delete-dbusers and force replica.my.cnf creation for all tools that did not have it [tools]
2016-12-08 §
18:48 <YuviPanda> restarted toolschecker on tools-checker-01 [tools]
2016-12-07 §
09:45 <YuviPanda> restart redis on tools-proxy-02 [tools]
09:32 <YuviPanda> cherry-pick https://gerrit.wikimedia.org/r/324210 and https://gerrit.wikimedia.org/r/324211 [tools]
09:29 <YuviPanda> clush -g k8s-worker -g k8s-master -g webproxy -b 'sudo puppet agent --disable "Deploying k8s change with alex"' [tools]
2016-12-06 §
00:36 <bd808> Updated toollabs-webservice to 0.31 on rest of cluster (T147350) [tools]
2016-12-05 §
23:19 <bd808> Updated toollabs-webservice to 0.31 on tools-bastion-02 (T147350) [tools]
22:55 <bd808> Updated jobutils to 1.17 on tools-mail (T147350) [tools]
22:53 <bd808> Updated jobutils to 1.17 on tools-precise-dev (T147350) [tools]
22:53 <bd808> Updated jobutils to 1.17 on tools-cron-01 (T147350) [tools]
22:52 <bd808> Updated jobutils to 1.17 on tools-bastion-03 (T147350) [tools]
22:52 <bd808> Updated jobutils to 1.17 on tools-bastion-02 (T147350) [tools]
16:53 <bd808> Terminated deprecated instances: "tools-exec-1201", "tools-exec-1202", "tools-exec-1203", "tools-exec-1205", "tools-exec-1206", "tools-exec-1207", "tools-exec-1208", "tools-exec-1209", "tools-exec-1210", "tools-exec-1211" (T151980) [tools]
16:50 <bd808> Released floating IPs from decommissioned tools-exec-12[01-11] instances [tools]
2016-11-30 §
23:06 <bd808> Removed tools-exec-12[00-11] from gridengine (T151980) [tools]
22:54 <bd808> Removed tools-exec-12[00-11] from @general hostgroup [tools]
15:17 <chasemp> restart coibot 'coibot.sh -o syslog.output -e syslog.errors -r yes' [tools]
05:20 <bd808> rescheduled continuous jobs on tools-exec-1210; 2 task queue jobs remain (T151980) [tools]
05:18 <bd808> drained tools-exec-1211 (T151980) [tools]
05:14 <bd808> drained tools-exec-1209 (T151980) [tools]