401-450 of 943 results (9ms)
2019-03-25 §
18:49 <bd808> All jobs still running on the Trusty job grid force deleted. [tools]
18:46 <bd808> All Trusty job grid queues marked as disabled. This should stop all new Trusty job submissions. [tools]
18:39 <bd808> Shutdown tools-cron-01.tools.eqiad.wmflabs (T217152) [tools]
15:27 <bd808> Copied all crontab files still on tools-cron-01 to tool's $HOME/crontab.trusty.save [tools]
02:34 <bd808> Disassociated floating IPs and deleted shutdown Trusty grid nodes tools-exec-14{33,34,35,36,37,38,39,40,41,42} (T217152) [tools]
02:26 <bd808> Deleted shutdown Trusty grid nodes tools-webgrid-lighttpd-14{20,21,22,24,25,26,27,28} (T217152) [tools]
2019-03-22 §
17:16 <andrewbogott> switching all instances to use ldap-ro.eqiad.wikimedia.org as both primary and secondary ldap server [tools]
15:56 <bd808> Rebooting tools-static-12 [tools]
03:09 <bstorm_> T217280 depooled and rebooted 15 other nodes. Entire stretch grid is in a good state for now. [tools]
02:31 <bstorm_> T217280 depooled and rebooted tools-sgeexec-0908 since it had no jobs but very high load from an NFS event that was no longer happening [tools]
02:09 <bstorm_> T217280 depooled and rebooted tools-sgewebgrid-lighttpd-0924 [tools]
00:39 <bstorm_> T217280 depooled and rebooted tools-sgewebgrid-lighttpd-0902 [tools]
2019-03-21 §
23:28 <bstorm_> T217280 depooled, reloaded and repooled tools-sgeexec-0938 [tools]
21:53 <bstorm_> T217280 rebooted and cleared "unknown status" from tools-sgeexec-0914 after depooling [tools]
21:51 <bstorm_> T217280 rebooted and cleared "unknown status" from tools-sgeexec-0909 after depooling [tools]
21:26 <bstorm_> T217280 cleared error state from a couple queues and rebooted tools-sgeexec-0901 and 04 to clear other issues related [tools]
2019-03-20 §
12:23 <arturo> depool and hard-reboot tools-sgewebgrid-generic-0904 due to extreme load. It doesn't respond to ssh [this one is valid] [tools]
12:22 <arturo> depool and hard-reboot tools-sgeexec-0938.eqiad.wmflabs due to extreme load. It doesn't respond to ssh [tools]
2019-03-18 §
18:43 <bd808> Rebooting tools-static-12 [tools]
18:25 <chicocvenancio> removing postStart hook for PWB update and restarting hub while gerrit.wikimedia.com is down [tools]
2019-03-17 §
23:41 <bd808> Cherry-picked https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/497210/ as a quick fix for T218494 [tools]
22:30 <bd808> Investigating strange system state on tools-bastion-03. [tools]
17:48 <bstorm_> T218514 rebooting tools-worker-1009 and 1012 [tools]
17:46 <bstorm_> depooling tools-worker-1009 and tools-worker-1012 for T218514 [tools]
17:13 <bstorm_> depooled and rebooting tools-worker-1018 [tools]
15:09 <andrewbogott> running 'killall dpkg and dpkg --configure -a' on all nodes to try to work around a race with initramfs [tools]
2019-03-15 §
14:40 <mutante> tools-sgebastion-07 - dpkg-reconfigure locales and adding Korean ko_KR.EUC-KR - T130532 [tools]
2019-03-14 §
23:52 <bd808> Disabled job queues and rescheduled continuous jobs away from tools-exec-14{21,22,23,24,25,26,27,28,29,30,31,32} (T217152) [tools]
23:28 <bd808> Deleted tools-bastion-05 (T217152) [tools]
22:30 <bd808> Removed obsolete submit hosts from Trusty grid config [tools]
22:20 <bd808> Removed tools-webgrid-lighttpd-142{0,1,2,5} from the grid and shutdown instances via horizon (T217152) [tools]
22:10 <bd808> Depooled tools-webgrid-lighttpd-142{0,1,2,5} (T217152) [tools]
21:55 <bd808> Removed submit host flag from tools-bastion-05.tools.eqiad.wmflabs, removed floating ip, and shutdown instance via horizon (T217152) [tools]
21:48 <bd808> Removed tools-exec-14{33,34,35,36,37,38,39,40,41,42} from the grid and shutdown instances via horizon (T217152) [tools]
21:19 <bd808> Killed jobs still running on tools-exec-14{33,34,35,36,37,38,39,40,41,42}.tools.eqiad.wmflabs 2 weeks after being depooled (T217152) [tools]
20:58 <bd808> Repooled tools-sgeexec-0941 following reboot [tools]
20:57 <bd808> Hard reboot of tools-sgeexec-0941 via horizon [tools]
20:54 <bd808> Depooled and rebooted tools-sgeexec-0941.tools.eqiad.wmflabs [tools]
20:53 <bd808> Repooled tools-sgeexec-0917 following reboot [tools]
20:52 <bd808> Hard reboot of tools-sgeexec-0917 via horizon [tools]
20:47 <bd808> Depooled and rebooted tools-sgeexec-0917 [tools]
20:44 <bd808> Repooled tools-sgeexec-0908 after reboot [tools]
20:36 <bd808> depooled and rebooted tools-sgeexec-0908 [tools]
2019-03-13 §
23:30 <bd808> Rebuilding stretch Kubernetes images [tools]
22:55 <bd808> Rebuilding jessie Kubernetes images [tools]
16:10 <bd808> Updated DNS for dev.tools.wmflabs.org to point to Stretch secondary bastion. This was missed on 2019-03-07 [tools]
00:22 <bd808> Raise web-memlimit for isbn tool to 6G for tomcat8 (T217406) [tools]
2019-03-11 §
15:53 <bd808> Manually started `service gridengine-master` on tools-sgegrid-master after reboot (T218038) [tools]
15:47 <bd808> Hard reboot of tools-sgegrid-master via Horizon UI (T218038) [tools]
15:42 <bd808> Rebooting tools-sgegrid-master (T218038) [tools]