51-100 of 2530 results (22ms)
2020-02-27 §
14:54 <bd808> Hard reboot tools-worker-1016. Direct virsh console unresponsive. Stuck in shutdown since 2020-01-22? [tools]
14:44 <bd808> Uncordoned tools-worker-1009.tools.eqiad.wmflabs [tools]
14:41 <bd808> Drained tools-worker-1032 [tools]
14:37 <bd808> Drained tools-worker-1033 [tools]
14:35 <bd808> Drained tools-worker-1034 [tools]
14:34 <bd808> Drained tools-worker-1035 [tools]
14:33 <bd808> Drained tools-worker-1036 [tools]
14:33 <bd808> Drained tools-worker-10{39,38,37} yesterday but did not !log [tools]
00:29 <bd808> Drained tools-worker-1009 for reboot (NFS flakey) [tools]
00:11 <bd808> Uncordoned tools-worker-1009.tools.eqiad.wmflabs [tools]
00:08 <bd808> Uncordoned tools-worker-1002.tools.eqiad.wmflabs [tools]
00:02 <bd808> Rebooting tools-worker-1002 [tools]
00:00 <bd808> Draining tools-worker-1002 to reboot for NFS problems [tools]
2020-02-26 §
23:42 <bd808> Drained tools-worker-1040 [tools]
23:41 <bd808> Cordoned tools-worker-10[16-40] in preparation for shrinking legacy Kubernetes cluster [tools]
23:12 <bstorm_> replacing all tool limit-ranges in the 2020 cluster with a lower cpu request version [tools]
22:29 <bstorm_> deleted pod maintain-kubeusers-6d9c45f4bc-5bqq5 to deploy new image [tools]
21:06 <bstorm_> deleting loads of stuck grid jobs [tools]
20:27 <jeh> rebooting tools-worker-[1008,1015,1021] [tools]
20:15 <bstorm_> rebooting tools-sgegrid-master because it actually had the permissions thing going on still [tools]
18:03 <bstorm_> downtimed toolschecker for nfs maintenance [tools]
2020-02-25 §
15:31 <bd808> `wmcs-k8s-enable-cluster-monitor toolschecker` [tools]
2020-02-23 §
00:40 <Krenair> T245932 [tools]
2020-02-21 §
16:02 <andrewbogott> moving tools-sgecron-01 to cloudvirt1022 [tools]
2020-02-20 §
14:49 <andrewbogott> moving tools-k8s-worker-19 and tools-k8s-worker-18 to cloudvirt1022 (as part of draining 1014) [tools]
00:04 <Krenair> Shut off tools-puppetmaster-01 - to be deleted in one week T245365 [tools]
2020-02-19 §
22:05 <Krenair> Project-wide hiera change to swap puppetmaster to tools-puppetmaster-02 T245365 [tools]
15:36 <bstorm_> setting 'puppetmaster: tools-puppetmaster-02.tools.eqiad.wmflabs' on tools-sgeexec-0942 to test new puppetmaster on grid T245365 [tools]
11:50 <arturo> fix invalid yaml format in horizon puppet prefix 'tools-k8s-haproxy' that prevented clean puppet run in the VMs [tools]
00:59 <bd808> Live hacked the "nginx-configuration" ConfigMap for T245426 (done several hours ago, but I forgot to !log it) [tools]
2020-02-18 §
23:26 <bstorm_> added tools-sgegrid-master.tools.eqiad1.wikimedia.cloud and tools-sgegrid-shadow.tools.eqiad1.wikimedia.cloud to gridengine admin host lists [tools]
09:50 <arturo> temporarily delete DNS zone tools.wmcloud.org to try re-creating it [tools]
2020-02-17 §
18:53 <arturo> T168677 created DNS TXT record _psl.toolforge.org. with value `https://github.com/publicsuffix/list/pull/970` [tools]
13:22 <arturo> relocating tools-sgewebgrid-lighttpd-0914 to cloudvirt1012 to spread same VMs across different hypervisors [tools]
2020-02-14 §
00:38 <bd808> Added tools-k8s-worker-35 to 2020 Kubernetes cluster (T244791) [tools]
00:34 <bd808> Added tools-k8s-worker-34 to 2020 Kubernetes cluster (T244791) [tools]
00:32 <bd808> Added tools-k8s-worker-33 to 2020 Kubernetes cluster (T244791) [tools]
00:29 <bd808> Added tools-k8s-worker-32 to 2020 Kubernetes cluster (T244791) [tools]
00:25 <bd808> Added tools-k8s-worker-31 to 2020 Kubernetes cluster (T244791) [tools]
00:25 <bd808> Added tools-k8s-worker-30 to 2020 Kubernetes cluster (T244791) [tools]
00:17 <bd808> Added tools-k8s-worker-29 to 2020 Kubernetes cluster (T244791) [tools]
00:15 <bd808> Added tools-k8s-worker-28 to 2020 Kubernetes cluster (T244791) [tools]
00:13 <bd808> Added tools-k8s-worker-27 to 2020 Kubernetes cluster (T244791) [tools]
00:07 <bd808> Added tools-k8s-worker-26 to 2020 Kubernetes cluster (T244791) [tools]
00:03 <bd808> Added tools-k8s-worker-25 to 2020 Kubernetes cluster (T244791) [tools]
2020-02-13 §
23:53 <bd808> Added tools-k8s-worker-24 to 2020 Kubernetes cluster (T244791) [tools]
23:50 <bd808> Added tools-k8s-worker-23 to 2020 Kubernetes cluster (T244791) [tools]
23:38 <bd808> Added tools-k8s-worker-22 to 2020 Kubernetes cluster (T244791) [tools]
21:35 <bd808> Deleted tools-sgewebgrid-lighttpd-092{1,2,3,4,5,6,7,8} & tools-sgewebgrid-generic-090{3,4} (T244791) [tools]
21:33 <bd808> Removed tools-sgewebgrid-lighttpd-092{1,2,3,4,5,6,7,8} & tools-sgewebgrid-generic-090{3,4} from grid engine config (T244791) [tools]