1-50 of 1742 results (17ms)
2019-03-27 §
23:35 <bstorm_> rebooted tools-paws-master-01 for NFS issue T219460 [tools]
14:45 <bstorm_> cleared several "E" state queues [tools]
12:26 <gtirloni> truncated exim4/paniclog on tools-sgewebgrid-lighttpd-0921 [tools]
12:25 <gtirloni> truncated exim4/paniclog on tools-sgecron-01 [tools]
12:15 <arturo> T218126 `aborrero@tools-sgegrid-master:~$ sudo qmod -d 'test@tools-sssd-sgeexec-test-2'` (and 1) [tools]
2019-03-26 §
22:00 <gtirloni> downtimed toolschecker [tools]
17:31 <arturo> T218126 create VM instances tools-sssd-sgeexec-test-[12] [tools]
00:26 <bd808> Deleted DNS record for login-trusty.tools.wmflabs.org [tools]
00:26 <bd808> Deleted DNS record for trusty-dev.tools.wmflabs.org [tools]
2019-03-25 §
21:21 <bd808> All Trusty grid engine hosts shutdown and deleted (T217152) [tools]
21:19 <bd808> Deleted tools-grid-{master,shadow} (T217152) [tools]
21:18 <bd808> Deleted tools-webgrid-lighttpd-14* (T217152) [tools]
20:55 <bstorm_> reboot tools-sgewebgrid-generic-0903 to clear up some issues [tools]
20:52 <bstorm_> rebooting tools-package-builder-02 due to lots of hung /usr/bin/lsof +c 15 -nXd DEL processes [tools]
20:51 <bd808> Deleted tools-webgrid-generic-14* (T217152) [tools]
20:49 <bd808> Deleted tools-exec-143* (T217152) [tools]
20:49 <bd808> Deleted tools-exec-142* (T217152) [tools]
20:48 <bd808> Deleted tools-exec-141* (T217152) [tools]
20:47 <bd808> Deleted tools-exec-140* (T217152) [tools]
20:43 <bd808> Deleted tools-cron-01 (T217152) [tools]
20:42 <bd808> Deleted tools-bastion-0{2,3} (T217152) [tools]
20:35 <bstorm_> rebooted tools-worker-1025 and tools-worker-1021 [tools]
19:59 <bd808> Shutdown tools-exec-143* (T217152) [tools]
19:51 <bd808> Shutdown tools-exec-142* (T217152) [tools]
19:47 <bstorm_> depooling tools-worker-1025.tools.eqiad.wmflabs because it's not responding and showing insane load [tools]
19:33 <bd808> Shutdown tools-exec-141* (T217152) [tools]
19:31 <bd808> Shutdown tools-bastion-0{2,3} (T217152) [tools]
19:19 <bd808> Shutdown tools-exec-140* (T217152) [tools]
19:12 <bd808> Shutdown tools-webgrid-generic-14* (T217152) [tools]
19:11 <bd808> Shutdown tools-webgrid-lighttpd-14* (T217152) [tools]
18:53 <bd808> Shutdown tools-grid-master (T217152) [tools]
18:53 <bd808> Shutdown tools-grid-shadow (T217152) [tools]
18:49 <bd808> All jobs still running on the Trusty job grid force deleted. [tools]
18:46 <bd808> All Trusty job grid queues marked as disabled. This should stop all new Trusty job submissions. [tools]
18:43 <arturo> icinga downtime tools-checker for 24h due to trusty grid shutdown [tools]
18:39 <bd808> Shutdown tools-cron-01.tools.eqiad.wmflabs (T217152) [tools]
15:27 <bd808> Copied all crontab files still on tools-cron-01 to tool's $HOME/crontab.trusty.save [tools]
02:34 <bd808> Disassociated floating IPs and deleted shutdown Trusty grid nodes tools-exec-14{33,34,35,36,37,38,39,40,41,42} (T217152) [tools]
02:26 <bd808> Deleted shutdown Trusty grid nodes tools-webgrid-lighttpd-14{20,21,22,24,25,26,27,28} (T217152) [tools]
2019-03-22 §
17:16 <andrewbogott> switching all instances to use ldap-ro.eqiad.wikimedia.org as both primary and secondary ldap server [tools]
16:12 <bstorm_> cleared errored out stretch grid queues [tools]
15:56 <bd808> Rebooting tools-static-12 [tools]
03:09 <bstorm_> T217280 depooled and rebooted 15 other nodes. Entire stretch grid is in a good state for now. [tools]
02:31 <bstorm_> T217280 depooled and rebooted tools-sgeexec-0908 since it had no jobs but very high load from an NFS event that was no longer happening [tools]
02:09 <bstorm_> T217280 depooled and rebooted tools-sgewebgrid-lighttpd-0924 [tools]
00:39 <bstorm_> T217280 depooled and rebooted tools-sgewebgrid-lighttpd-0902 [tools]
2019-03-21 §
23:28 <bstorm_> T217280 depooled, reloaded and repooled tools-sgeexec-0938 [tools]
21:53 <bstorm_> T217280 rebooted and cleared "unknown status" from tools-sgeexec-0914 after depooling [tools]
21:51 <bstorm_> T217280 rebooted and cleared "unknown status" from tools-sgeexec-0909 after depooling [tools]
21:26 <bstorm_> T217280 cleared error state from a couple queues and rebooted tools-sgeexec-0901 and 04 to clear other issues related [tools]