651-700 of 943 results (14ms)
2018-01-06 §
00:05 <madhuvishy> Drain and cordon tools-worker-1001 (for debugging the dns outage) [tools]
2018-01-03 §
15:38 <bd808> Forced Puppet run on tools-services-01 [tools]
11:29 <arturo> deploy https://gerrit.wikimedia.org/r/#/c/401716/ and https://gerrit.wikimedia.org/r/394101 using clush [tools]
2017-12-31 §
02:00 <bd808> Killed some pwb.py and qacct processes running on tools-bastion-03 [tools]
2017-12-21 §
17:57 <bd808> PAWS: deleted hub-deployment pod stuck in crashloopbackoff [tools]
17:30 <bd808> PAWS: deleting hub-deployment pod. Lots of "Connection pool is full" warnings in pod logs [tools]
2017-12-13 §
00:59 <madhuvishy> Cordon and Drain tools-worker-1016 [tools]
2017-12-11 §
19:32 <bd808> git gc on tools-static-11; --aggressive was killed by system (T182604) [tools]
17:07 <bd808> git gc --aggressive on tools-static-11 (T182604) [tools]
2017-11-30 §
23:23 <bd808> Hard reboot of tools-bastion-03 via Horizon [tools]
2017-11-17 §
21:33 <valhallasw`cloud> also g-w'ed those files, and sent emails to all the affected users [tools]
2017-11-16 §
16:50 <bd808> Force upgraded nginx on tools-elastic-* [tools]
2017-11-15 §
15:48 <chasemp> kill tools.powow on bastion-03 for hammering IO and making bastion unusable [tools]
2017-11-07 §
01:21 <bd808> Removed all non-directory files from /home (via labstore1004 direct access) [tools]
2017-11-06 §
18:30 <bd808> Load on tools-bastion-03 down to 0.72 from 17.47 after killing a bunch of local processes that should have been running on the job grid instead [tools]
2017-11-05 §
23:48 <bd808> Cleaned up 2 huge /tmp files left by tools.croptool (~6.5G) [tools]
23:44 <bd808> Cleaned up 109 files owned by tools.rezabot on tools-webgrid-lighttpd-1428 with `sudo find /tmp -user tools.rezabot -exec rm {} \+` [tools]
23:37 <bd808> Cleaned up 955 files owned by tools.wsexport on tools-webgrid-lighttpd-1428 with `sudo find /tmp -user tools.wsexport -exec rm {} \+` [tools]
2017-11-03 §
21:19 <bd808> Deployed misctools 1.26 (T156174) [tools]
2017-11-02 §
16:15 <bd808> Restarted nslcd on tools-bastion-03 [tools]
2017-10-31 §
16:50 <bd808> tools-bastion-03 (tools-login, login.tools) is overloaded [tools]
2017-10-23 §
14:49 <chasemp> wall message and scheduled reboot in 5m for bastion-03 [tools]
2017-10-18 §
21:36 <chasemp> stop basebot -- it is going crazy and spamming email w/ failing to log to error.log. Need to figure out how to notify but it's clearly in a failure loop. [tools]
2017-10-12 §
16:57 <bd808> Rebuilding all Kubernetes Docker images to include toollabs-webservice 0.38 [tools]
16:53 <bd808> Upgraded toollabs-webservice to 0.38 [tools]
2017-10-06 §
15:33 <bd808> Upgrade jobutils to 1.25 (T177614) [tools]
00:27 <bd808> Updated misctools to 1.24 [tools]
2017-10-05 §
22:47 <bd808> Updated misctools to 1.23 [tools]
22:42 <bd808> Updated jobutils to 1.23 [tools]
2017-10-03 §
19:30 <bd808> `kubectl --namespace=prod delete pod --all` on tools-paws-master-01 [tools]
2017-09-19 §
13:25 <chasemp> wall Bastion disk is full and needs attention and reboot in 60 [tools]
2017-09-18 §
18:02 <bd808> Updated PHP5.6 images for Kubernetes (T172358) [tools]
2017-09-13 §
15:34 <bd808> Running inbound message purge via clush to @tools-exec [tools]
15:15 <bd808> Running outbound message purge via clush to @tools-exec [tools]
13:57 <bd808> apt-get install nginx-common on tools-static-1[01] [tools]
13:31 <bd808> static down due to apparent nginx package upgrade/config change [tools]
02:10 <bd808> Really disabled puppet on tools-mail [tools]
01:51 <bd808> Nuked all messages in the exim spool on tools-mail [tools]
01:09 <bd808> Removed user WiktCAPT from project [tools]
00:55 <bd808> Archived and then purged /var/spool/exim4/input on tools-mail [tools]
00:47 <bd808> Archived and then purged /var/spool/exim4/msglog on tools-mail [tools]
00:43 <bd808> Stopped exim on tools-mail [tools]
00:43 <bd808> Disabled puppet on tools-mail [tools]
2017-08-31 §
20:33 <madhuvishy> Updated certs and ran puppet, restarted nginx on tools-proxy-* and tools-static-* (T174611) [tools]
20:24 <madhuvishy> Disabling puppet on tools-proxy-* and tools-static-* for star.wmflabs.org SSL cert update (T174611) [tools]
20:23 <madhuvishy> Disabling puppet on tools-proxy-* and tools-static-* for star.wmflabs.org SSL cert update [tools]
2017-08-24 §
19:59 <bd808> restarted nslcd and nscd on tools-bastion-03 [tools]
19:59 <bd808> restarted nslcd and nscd on tools-bastion-02 [tools]
2017-08-03 §
00:47 <bd808> tools-bastion-03 not usably responsive to interactive commands; will reboot [tools]
00:00 <bd808> Restarted kube-proxy service on bastion-03 [tools]