901-950 of 1864 results (19ms)
2017-11-30 §
23:06 <chasemp> rebooting login.tools.wmflabs.org due to overload [tools]
2017-11-20 §
20:34 <chasemp> backup crons tools-cron-01:/var/spool/cron# cp -Rp crontabs/ /root/20112017/ [tools]
00:52 <andrewbogott> cherry-picking https://gerrit.wikimedia.org/r/#/c/392172/ onto the tools puppetmaster [tools]
2017-11-17 §
21:33 <valhallasw`cloud> also g-w'ed those files, and sent emails to all the affected users [tools]
21:17 <valhallasw`cloud> chmod o-w'ed a bunch of files reported by Dispenser; writing emails to the owners about this [tools]
2017-11-16 §
17:40 <chasemp> tools-clushmaster-01:~$ clush -w @all 'sudo puppet agent --enable && sudo puppet agent --test && sudo unattended-upgrades -d' [tools]
16:50 <bd808> Force upgraded nginx on tools-elastic-* [tools]
16:37 <chasemp> reboot tools-checker-01 [tools]
15:17 <chasemp> disable puppet [tools]
2017-11-15 §
22:48 <madhuvishy> Rebooted tools-paws-worker-1017 [tools]
15:53 <chasemp> reboot bastion-03 [tools]
15:48 <chasemp> kill tools.powow on bastion-03 for hammering IO and making bastion unusable [tools]
2017-11-07 §
01:21 <bd808> Removed all non-directory files from /home (via labstore1004 direct access) [tools]
2017-11-06 §
18:30 <bd808> Load on tools-bastion-03 down to 0.72 from 17.47 after killing a bunch of local processes that should have been running on the job grid instead [tools]
2017-11-05 §
23:48 <bd808> Cleaned up 2 huge /tmp files left by tools.croptool (~6.5G) [tools]
23:44 <bd808> Cleaned up 109 files owned by tools.rezabot on tools-webgrid-lighttpd-1428 with `sudo find /tmp -user tools.rezabot -exec rm {} \+` [tools]
23:37 <bd808> Cleaned up 955 files owned by tools.wsexport on tools-webgrid-lighttpd-1428 with `sudo find /tmp -user tools.wsexport -exec rm {} \+` [tools]
2017-11-03 §
21:19 <bd808> Deployed misctools 1.26 (T156174) [tools]
2017-11-02 §
16:15 <bd808> Restarted nslcd on tools-bastion-03 [tools]
2017-11-01 §
07:11 <madhuvishy> Clear nscd cache across all projects post labsdb dns switchover T179464 [tools]
07:11 <madhuvishy> Clear nscd cache across all projects post labsdb dns switchover [tools]
2017-10-31 §
16:50 <bd808> tools-bastion-03 (tools-login, login.tools) is overloaded [tools]
2017-10-30 §
17:35 <madhuvishy> Clear dns caches across tools hosts `sudo nscd -i hosts` [tools]
16:08 <arturo> depool tools-exec-1401.tools.eqiad.wmflabs [tools]
2017-10-24 §
18:09 <madhuvishy> Disable puppet on tools-package-builder-01 temporarily (T178920) [tools]
13:22 <chasemp> start admin webservice [tools]
13:22 <chasemp> stop admin webservice [tools]
2017-10-23 §
14:49 <chasemp> wall message and scheduled reboot in 5m for bastion-03 [tools]
2017-10-18 §
21:36 <chasemp> stop basebot -- it is going crazy and spamming email w/ failing to log to error.log. Need to figure out how to notify but it's clearly in a failure loop. [tools]
14:04 <chasemp> add strephit creds to elasticsearch per T178310 [tools]
2017-10-12 §
16:57 <bd808> Rebuilding all Kubernetes Docker images to include toollabs-webservice 0.38 [tools]
16:53 <bd808> Upgraded toollabs-webservice to 0.38 [tools]
2017-10-06 §
15:33 <bd808> Upgrade jobutils to 1.25 (T177614) [tools]
00:27 <bd808> Updated misctools to 1.24 [tools]
2017-10-05 §
22:47 <bd808> Updated misctools to 1.23 [tools]
22:42 <bd808> Updated jobutils to 1.23 [tools]
15:46 <chasemp> tools-bastion-03 has tons of local tools running long lived NFS intensive processes. I'm rebooting rather than playing whackamole. [tools]
2017-10-03 §
19:30 <bd808> `kubectl --namespace=prod delete pod --all` on tools-paws-master-01 [tools]
2017-10-01 §
21:46 <madhuvishy> Cold migrating tools-clushmaster-01 from labvirt1015 to labvirt1017 [tools]
2017-09-29 §
19:49 <andrewbogott> migration tools-clushmaster-01 to labvirt1015 [tools]
2017-09-25 §
15:14 <andrewbogott> rebooting tools-paws-worker-1006 since I can't access it [tools]
14:57 <chasemp> OS_TENANT_NAME=tools openstack server reboot 2c0cf363-c7c3-42ad-94bd-e586f2492321 (unresponsive) [tools]
2017-09-20 §
16:52 <madhuvishy> apt-get install --only-upgrade apache2; service apache2 restart on tools-puppetmaster-01 [tools]
2017-09-19 §
15:22 <chasemp> tools-clushmaster-01:~$ clush -f 5 -g all 'sudo puppet agent --test' [tools]
13:39 <chasemp> bastion-03 someone dropped 8.6G in /tmp which is /not/ seemingly on a temp file system [tools]
13:25 <chasemp> wall Bastion disk is full and needs attention and reboot in 60 [tools]
2017-09-18 §
18:02 <bd808> Updated PHP5.6 images for Kubernetes (T172358) [tools]
2017-09-13 §
15:34 <bd808> Running inbound message purge via clush to @tools-exec [tools]
15:15 <bd808> Running outbound message purge via clush to @tools-exec [tools]
13:57 <bd808> apt-get install nginx-common on tools-static-1[01] [tools]