201-250 of 2708 results (11ms)
2020-02-28 §
04:13 <bd808> Joined tools-k8s-worker-43 to 2020 Kubernetes cluster [tools]
04:12 <bd808> Joined tools-k8s-worker-42 to 2020 Kubernetes cluster [tools]
04:10 <bd808> Joined tools-k8s-worker-41 to 2020 Kubernetes cluster [tools]
04:09 <bd808> Joined tools-k8s-worker-40 to 2020 Kubernetes cluster [tools]
04:08 <bd808> Joined tools-k8s-worker-39 to 2020 Kubernetes cluster [tools]
04:07 <bd808> Joined tools-k8s-worker-38 to 2020 Kubernetes cluster [tools]
04:06 <bd808> Joined tools-k8s-worker-37 to 2020 Kubernetes cluster [tools]
03:48 <bd808> Joined tools-k8s-worker-36 to 2020 Kubernetes cluster [tools]
00:50 <bstorm_> rebuilt all docker images to include webservice 0.64 [tools]
2020-02-27 §
23:27 <bstorm_> installed toollabs-webservice 0.64 on the bastions [tools]
23:23 <bstorm_> pushed toollabs-webservice version 0.64 to all toolforge repos [tools]
21:03 <jeh> add reindex service account to elasticsearch for data migration T236606 [tools]
20:57 <bstorm_> upgrading toollabs-webservice to stretch-toolsbeta version for jdk8:testing image only [tools]
20:19 <jeh> update elasticsearch VPS security group to allow toolsbeta-elastic7-1 access on tcp 80 T236606 [tools]
18:53 <bstorm_> hard rebooted a rather stuck tools-sgecron-01 [tools]
18:20 <bd808> Building tools-k8s-worker-[36-55] [tools]
17:56 <bd808> Deleted instances tools-worker-10[21-40] [tools]
16:14 <bd808> Decommissioning tools-worker-10[21-40] [tools]
16:02 <bd808> Drained tools-worker-1021 [tools]
15:51 <bd808> Drained tools-worker-1022 [tools]
15:44 <bd808> Drained tools-worker-1023 (there is no tools-worker-1024) [tools]
15:39 <bd808> Drained tools-worker-1025 [tools]
15:39 <bd808> Drained tools-worker-1026 [tools]
15:11 <bd808> Drained tools-worker-1027 [tools]
15:09 <bd808> Drained tools-worker-1028 (there is no tools-worker-1029) [tools]
15:07 <bd808> Drained tools-worker-1030 [tools]
15:06 <bd808> Uncordoned tools-worker-10[16-20]. Was over optimistic about repacking legacy Kubernetes cluster into 15 instances. Will keep 20 for now. [tools]
15:00 <bd808> Drained tools-worker-1031 [tools]
14:54 <bd808> Hard reboot tools-worker-1016. Direct virsh console unresponsive. Stuck in shutdown since 2020-01-22? [tools]
14:44 <bd808> Uncordoned tools-worker-1009.tools.eqiad.wmflabs [tools]
14:41 <bd808> Drained tools-worker-1032 [tools]
14:37 <bd808> Drained tools-worker-1033 [tools]
14:35 <bd808> Drained tools-worker-1034 [tools]
14:34 <bd808> Drained tools-worker-1035 [tools]
14:33 <bd808> Drained tools-worker-1036 [tools]
14:33 <bd808> Drained tools-worker-10{39,38,37} yesterday but did not !log [tools]
00:29 <bd808> Drained tools-worker-1009 for reboot (NFS flakey) [tools]
00:11 <bd808> Uncordoned tools-worker-1009.tools.eqiad.wmflabs [tools]
00:08 <bd808> Uncordoned tools-worker-1002.tools.eqiad.wmflabs [tools]
00:02 <bd808> Rebooting tools-worker-1002 [tools]
00:00 <bd808> Draining tools-worker-1002 to reboot for NFS problems [tools]
2020-02-26 §
23:42 <bd808> Drained tools-worker-1040 [tools]
23:41 <bd808> Cordoned tools-worker-10[16-40] in preparation for shrinking legacy Kubernetes cluster [tools]
23:12 <bstorm_> replacing all tool limit-ranges in the 2020 cluster with a lower cpu request version [tools]
22:29 <bstorm_> deleted pod maintain-kubeusers-6d9c45f4bc-5bqq5 to deploy new image [tools]
21:06 <bstorm_> deleting loads of stuck grid jobs [tools]
20:27 <jeh> rebooting tools-worker-[1008,1015,1021] [tools]
20:15 <bstorm_> rebooting tools-sgegrid-master because it actually had the permissions thing going on still [tools]
18:03 <bstorm_> downtimed toolschecker for nfs maintenance [tools]
2020-02-25 §
15:31 <bd808> `wmcs-k8s-enable-cluster-monitor toolschecker` [tools]