51-100 of 1874 results (10ms)
2019-04-10 §
18:52 <bstorm_> depooled and rebooted tools-sgeexec-0929 because systemd was in a weird state [tools]
18:46 <bstorm_> depooled and rebooted tools-sgewebgrid-lighttpd-0913 because high load was caused by ancient lsof processes [tools]
14:49 <bstorm_> cleared E state from 5 queues [tools]
13:06 <arturo> T218126 hard reboot tools-sgeexec-0906 [tools]
12:31 <arturo> T218126 hard reboot tools-sgeexec-0926 [tools]
12:27 <arturo> T218126 hard reboot tools-sgeexec-0925 [tools]
12:06 <arturo> T218126 hard reboot tools-sgeexec-0901 [tools]
11:55 <arturo> T218126 hard reboot tools-sgeexec-0924 [tools]
11:47 <arturo> T218126 hard reboot tools-sgeexec-0921 [tools]
11:23 <arturo> T218126 hard reboot tools-sgeexec-0940 [tools]
11:03 <arturo> T218126 hard reboot tools-sgeexec-0928 [tools]
10:49 <arturo> T218126 hard reboot tools-sgeexec-0923 [tools]
10:43 <arturo> T218126 hard reboot tools-sgeexec-0915 [tools]
10:27 <arturo> T218126 hard reboot tools-sgeexec-0935 [tools]
10:19 <arturo> T218126 hard reboot tools-sgeexec-0914 [tools]
10:01 <arturo> T218126 hard reboot tools-sgeexec-0907 [tools]
09:40 <arturo> T218126 hard reboot tools-sgeexec-0918 [tools]
09:27 <arturo> T218126 hard reboot tools-sgeexec-0932 [tools]
09:26 <arturo> T218216 hard reboot tools-sgeexec-0932 [tools]
09:04 <arturo> T218216 add `profile::ldap::client::labs::client_stack: sssd` to prefix puppet for sge-exec nodes [tools]
09:03 <arturo> T218216 do a controlled rollover of sssd, depooling sgeexec nodes, reboot and repool [tools]
08:39 <arturo> T218216 disable puppet in all tools-sgeexec-XXXX nodes for controlled sssd rollout [tools]
00:32 <andrewbogott> migrating tools-worker-1022, 1023, 1025, 1026 to eqiad1-r [tools]
2019-04-09 §
22:04 <bstorm_> added the new region on port 80 to the elasticsearch security group for stashbot [tools]
20:43 <andrewbogott> moving tools-worker-1018, 1019, 1020, 1021 to eqiad1-r [tools]
20:04 <andrewbogott> moving tools-k8s-etcd-03 to eqiad1-r [tools]
19:54 <andrewbogott> moving tools-flannel-etcd-02 to eqiad1-r [tools]
18:36 <andrewbogott> moving tools-worker-1016, tools-worker-1017 to eqiad1-r [tools]
18:05 <andrewbogott> migrating tools-k8s-etcd-02 to eqiad1-r [tools]
18:00 <andrewbogott> migrating tools-flannel-etcd-01 to eqiad1-r [tools]
17:36 <andrewbogott> moving tools-worker-1014, tools-worker-1015 to eqiad1-r [tools]
17:05 <andrewbogott> migrating tools-k8s-etcd-01 to eqiad1-r [tools]
15:56 <andrewbogott> moving tools-worker-1012, tools-worker-1013 to eqiad1-r [tools]
14:56 <bstorm_> cleared 4 queues on gridengine of E status (ldap again) [tools]
14:07 <andrewbogott> moving tools-worker-1010, tools-worker-1011, tools-worker-1001 to eqiad1-r [tools]
03:48 <andrewbogott> moving tools-worker-1008 and tools-worker-1009 to eqiad1-r [tools]
02:07 <bstorm_> reloaded ferm on tools-flannel-etcd-0[1-3] to get the k8s node moves to register [tools]
2019-04-08 §
22:36 <andrewbogott> moving tools-worker-1006 and tools-worker-1007 to eqiad1-r [tools]
20:03 <andrewbogott> moving tools-worker-1003 and tools-worker-1004 to eqiad1-r [tools]
2019-04-07 §
16:54 <zhuyifei1999_> tools-sgeexec-0928 unresponsive since around 22 UTC. No data on Graphite. Can't ssh in even as root. Hard rebooting via Horizon [tools]
01:06 <bstorm_> cleared E state from 6 queues [tools]
2019-04-05 §
15:44 <bstorm_> cleared E state from two exec queues [tools]
2019-04-04 §
21:21 <bd808> Uncordoned tools-worker-1013.tools.eqiad.wmflabs after reboot and forced puppet run [tools]
20:53 <bd808> Rebooting tools-worker-1013 [tools]
20:50 <bd808> Draining tools-worker-1013.tools.eqiad.wmflabs [tools]
20:29 <bd808> Released floating IP and deleted instance tools-checker-01 via Horizon [tools]
20:28 <bd808> Shutdown tools-checker-01 via Horizon [tools]
20:17 <bd808> Repooled tools-webgrid-lighttpd-0906 after reboot, apt-get dist-upgrade, and forced puppet run [tools]
20:13 <bd808> Hard reboot of tools-sgewebgrid-lighttpd-0906 via Horizon [tools]
20:09 <bd808> Repooled tools-webgrid-lighttpd-0912 after reboot, apt-get dist-upgrade, and forced puppet run [tools]