101-150 of 2885 results (19ms)
2020-10-06 §
18:51 <andrewbogott> uncordoned tools-k8s-worker-52 [tools]
18:40 <andrewbogott> draining and cordoning tools-k8s-worker-52 and tools-k8s-worker-38 for ceph migration [tools]
2020-10-02 §
21:09 <bstorm> rebooting tools-k8s-worker-70 because it seems to be unable to recover from an old NFS disconnect [tools]
17:37 <andrewbogott> stopping tools-prometheus-03 to attempt a snapshot [tools]
16:03 <bstorm> shutting down tools-prometheus-04 to try to fsck the disk [tools]
2020-10-01 §
21:39 <andrewbogott> migrating tools-proxy-06 to ceph [tools]
21:35 <andrewbogott> moving k8s.tools.eqiad1.wikimedia.cloud from 172.16.0.99 (toolsbeta-test-k8s-haproxy-1) to 172.16.0.108 (toolsbeta-test-k8s-haproxy-2) in anticipation of downtime for haproxy-1 tomorrow [tools]
2020-09-30 §
18:34 <andrewbogott> repooling tools-sgeexec-0918 [tools]
18:29 <andrewbogott> depooling tools-sgeexec-0918 so I can reboot cloudvirt1036 [tools]
2020-09-23 §
21:38 <bstorm> ran an 'apt clean' across the fleet to get ahead of the new locale install [tools]
2020-09-18 §
19:41 <andrewbogott> repooling tools-k8s-worker-30, 33, 34, 57, 60 [tools]
19:04 <andrewbogott> depooling tools-k8s-worker-30, 33, 34, 57, 60 [tools]
19:02 <andrewbogott> repooling tools-k8s-worker-41, 43, 44, 47, 48, 49, 50, 51 [tools]
17:48 <andrewbogott> depooling tools-k8s-worker-41, 43, 44, 47, 48, 49, 50, 51 [tools]
17:47 <andrewbogott> repooling tools-k8s-worker-31, 32, 36, 39, 40 [tools]
16:40 <andrewbogott> depooling tools-k8s-worker-31, 32, 36, 39, 40 [tools]
16:38 <andrewbogott> repooling tools-sgewebgrid-lighttpd-0914, tools-sgewebgrid-generic-0902, tools-sgewebgrid-lighttpd-0919, tools-sgewebgrid-lighttpd-0918 [tools]
16:10 <andrewbogott> depooling tools-sgewebgrid-lighttpd-0914, tools-sgewebgrid-generic-0902, tools-sgewebgrid-lighttpd-0919, tools-sgewebgrid-lighttpd-0918 [tools]
13:54 <andrewbogott> repooling tools-sgeexec-0913, tools-sgeexec-0915, tools-sgeexec-0916 [tools]
13:50 <andrewbogott> depooling tools-sgeexec-0913, tools-sgeexec-0915, tools-sgeexec-0916 for flavor update [tools]
01:20 <andrewbogott> repooling tools-sgeexec-0901, tools-sgeexec-0905, tools-sgeexec-0910, tools-sgeexec-0911, tools-sgeexec-0912 after flavor update [tools]
01:11 <andrewbogott> depooling tools-sgeexec-0901, tools-sgeexec-0905, tools-sgeexec-0910, tools-sgeexec-0911, tools-sgeexec-0912 for flavor update [tools]
01:08 <andrewbogott> repooling tools-sgeexec-0917, tools-sgeexec-0918, tools-sgeexec-0919, tools-sgeexec-0920 after flavor update [tools]
01:00 <andrewbogott> depooling tools-sgeexec-0917, tools-sgeexec-0918, tools-sgeexec-0919, tools-sgeexec-0920 for flavor update [tools]
00:58 <andrewbogott> repooling tools-sgeexec-0942, tools-sgeexec-0947, tools-sgeexec-0950, tools-sgeexec-0951, tools-sgeexec-0952 after flavor update [tools]
00:49 <andrewbogott> depooling tools-sgeexec-0942, tools-sgeexec-0947, tools-sgeexec-0950, tools-sgeexec-0951, tools-sgeexec-0952 for flavor update [tools]
2020-09-17 §
21:56 <bd808> Built and deployed tools-manifest v0.22 (T263190) [tools]
21:55 <bd808> Built and deployed tools-manifest v0.22 (T169695) [tools]
20:34 <bd808> Live hacked "--backend=gridengine" into webservicemonitor on tools-sgecron-01 (T263190) [tools]
20:21 <bd808> Restarted webservicemonitor on tools-sgecron-01.tools.eqiad.wmflabs [tools]
20:09 <andrewbogott> I didn't actually depool tools-sgeexec-0942, tools-sgeexec-0947, tools-sgeexec-0950, tools-sgeexec-0951, tools-sgeexec-0952 because there was some kind of brief outage just now [tools]
19:58 <andrewbogott> depooling tools-sgeexec-0942, tools-sgeexec-0947, tools-sgeexec-0950, tools-sgeexec-0951, tools-sgeexec-0952 for flavor update [tools]
19:55 <andrewbogott> repooling tools-k8s-worker-61,62,64,65,67,68,69 for flavor update [tools]
19:29 <andrewbogott> depooling tools-k8s-worker-61,62,64,65,67,68,69 for flavor update [tools]
15:38 <andrewbogott> repooling tools-k8s-worker-70 and tools-k8s-worker-66 after flavor remapping [tools]
15:34 <andrewbogott> depooling tools-k8s-worker-70 and tools-k8s-worker-66 for flavor remapping [tools]
15:30 <andrewbogott> repooling tools-sgeexec-0909, 0908, 0907, 0906, 0904 [tools]
15:21 <andrewbogott> depooling tools-sgeexec-0909, 0908, 0907, 0906, 0904 for flavor remapping [tools]
13:55 <andrewbogott> depooled tools-sgewebgrid-lighttpd-0917 and tools-sgewebgrid-lighttpd-0920 [tools]
13:55 <andrewbogott> repooled tools-sgeexec-0937 after move to ceph [tools]
13:45 <andrewbogott> depooled tools-sgeexec-0937 for move to ceph [tools]
2020-09-16 §
23:20 <andrewbogott> repooled tools-sgeexec-0941 and tools-sgeexec-0939 for move to ceph [tools]
23:03 <andrewbogott> depooled tools-sgeexec-0941 and tools-sgeexec-0939 for move to ceph [tools]
23:02 <andrewbogott> uncordoned tools-k8s-worker-58, tools-k8s-worker-56, tools-k8s-worker-42 for migration to ceph [tools]
22:29 <andrewbogott> draining tools-k8s-worker-58, tools-k8s-worker-56, tools-k8s-worker-42 for migration to ceph [tools]
17:37 <andrewbogott> service gridengine-master restart on tools-sgegrid-master [tools]
2020-09-10 §
15:37 <arturo> hard-rebooting tools-proxy-05 [tools]
15:33 <arturo> rebooting tools-proxy-05 to try flushing local DNS caches [tools]
15:25 <arturo> detected missing DNS record for k8s.tools.eqiad1.wikimedia.cloud which means the k8s cluster is down [tools]
10:22 <arturo> enabling ingress dedicated worker nodes in the k8s cluster (T250172) [tools]