201-250 of 1864 results (10ms)
2019-03-14 §
22:10 <bd808> Depooled tools-webgrid-lighttpd-142{0,1,2,5} (T217152) [tools]
21:55 <bd808> Removed submit host flag from tools-bastion-05.tools.eqiad.wmflabs, removed floating ip, and shutdown instance via horizon (T217152) [tools]
21:48 <bd808> Removed tools-exec-14{33,34,35,36,37,38,39,40,41,42} from the grid and shutdown instances via horizon (T217152) [tools]
21:38 <gtirloni> rebooted tools-sgewebgrid-generic-0904 (T218341) [tools]
21:32 <gtirloni> rebooted tools-exec-1020 (T218341) [tools]
21:23 <gtirloni> rebooted tools-sgeexec-0919, tools-sgeexec-0934, tools-worker-1018 (T218341) [tools]
21:19 <bd808> Killed jobs still running on tools-exec-14{33,34,35,36,37,38,39,40,41,42}.tools.eqiad.wmflabs 2 weeks after being depooled (T217152) [tools]
20:58 <bd808> Repooled tools-sgeexec-0941 following reboot [tools]
20:57 <bd808> Hard reboot of tools-sgeexec-0941 via horizon [tools]
20:54 <bd808> Depooled and rebooted tools-sgeexec-0941.tools.eqiad.wmflabs [tools]
20:53 <bd808> Repooled tools-sgeexec-0917 following reboot [tools]
20:52 <bd808> Hard reboot of tools-sgeexec-0917 via horizon [tools]
20:47 <bd808> Depooled and rebooted tools-sgeexec-0917 [tools]
20:44 <bd808> Repooled tools-sgeexec-0908 after reboot [tools]
20:36 <bd808> depooled and rebooted tools-sgeexec-0908 [tools]
19:08 <gtirloni> rebooted tools-worker-1028 (T218341) [tools]
19:08 <gtirloni> rebooted tools-sgewebgrid-lighttpd-0914 (T218341) [tools]
19:07 <gtirloni> rebooted tools-sgewebgrid-lighttpd-0914 [tools]
18:13 <gtirloni> drained tools-worker-1028 for reboot (processes in D state) [tools]
2019-03-13 §
23:30 <bd808> Rebuilding stretch Kubernetes images [tools]
22:55 <bd808> Rebuilding jessie Kubernetes images [tools]
17:11 <bstorm_> specifically rebooted SGE cron server tools-sgecron-01 [tools]
17:10 <bstorm_> rebooted cron server [tools]
16:10 <bd808> Updated DNS for dev.tools.wmflabs.org to point to Stretch secondary bastion. This was missed on 2019-03-07 [tools]
12:33 <arturo> reboot tools-sgebastion-08 (T215154) [tools]
12:17 <arturo> reboot tools-sgebastion-07 (T215154) [tools]
11:53 <arturo> enable puppet in tools-sgebastion-07 (T215154) [tools]
11:20 <arturo> disable puppet in tools-sgebastion-07 for testing T215154 [tools]
05:07 <bstorm_> re-enabled puppet for tools-sgebastion-07 [tools]
04:59 <bstorm_> disabled puppet for a little bit on tools-bastion-07 [tools]
00:22 <bd808> Raise web-memlimit for isbn tool to 6G for tomcat8 (T217406) [tools]
2019-03-11 §
15:53 <bd808> Manually started `service gridengine-master` on tools-sgegrid-master after reboot (T218038) [tools]
15:47 <bd808> Hard reboot of tools-sgegrid-master via Horizon UI (T218038) [tools]
15:42 <bd808> Rebooting tools-sgegrid-master (T218038) [tools]
14:49 <gtirloni> deleted tools-webgrid-lighttpd-1419 [tools]
00:53 <bd808> Re-enabled 13 queue instances that had been disabled by LDAP failures during job initialization (T217280) [tools]
2019-03-10 §
22:36 <gtirloni> increased nscd group TTL from 60 to 300sec [tools]
2019-03-08 §
19:48 <andrewbogott> repooling tools-exec-1430 and tools-sgeexec-0905 to compare ldap usage [tools]
19:21 <andrewbogott> depooling tools-exec-1430 and tools-sgeexec-0905 to compare ldap usage [tools]
17:49 <bd808> Re-enabled 4 queue instances that had been disabled by LDAP failures during job initialization (T217280) [tools]
00:30 <bd808> DNS record created for trusty-dev.tools.wmflabs.org (Trusty secondary bastion) [tools]
2019-03-07 §
23:31 <bd808> Updated DNS to point login.tools.wmflabs.org at 185.15.56.48 (Stretch bastion) [tools]
04:15 <bd808> Killed 3 orphan processes on Trusty grid [tools]
04:01 <bd808> Cleared error state on a large number of Stretch grid queues which had been disabled by LDAP and/or NFS hiccups (T217280) [tools]
00:49 <zhuyifei1999_> clushed misctools 1.37 upgrade on @bastion,@cron,@bastion-stretch T217406 [tools]
00:38 <zhuyifei1999_> published misctools 1.37 T217406 [tools]
00:34 <zhuyifei1999_> begin building misctools 1.37 using debuild T217406 [tools]
2019-03-06 §
13:57 <gtirloni> fixed SSH warnings in tools-clushmaster-02 [tools]
2019-03-04 §
19:07 <bstorm_> umounted /mnt/nfs/dumps-labstore1006.wikimedia.org for T217473 [tools]
14:05 <gtirloni> rebooted tools-docker-registry-{03,04}, tools-puppetmaster-02 and tools-puppetdb-01 (load avg >45, not accessible) [tools]