|
2020-02-28
§
|
| 16:36 |
<bstorm_> |
rebooting k8s workers 1-35 on the 2020 cluster to clear a strange nologin condition that has been there since the NFS maintenance |
[tools] |
| 16:14 |
<bstorm_> |
rebooted tools-k8s-worker-7 to clear some puppet issues |
[tools] |
| 16:00 |
<bd808> |
Devoicing stashbot in #wikimedia-cloud to reduce irc spam while migrating tools to 2020 Kubernetes cluster |
[tools] |
| 15:28 |
<jeh> |
create OpenStack server group tools-elastic with anti-affinty policy enabled T236606 |
[tools] |
| 15:09 |
<jeh> |
create 3 new elasticsearch VMs tools-elastic-[1,2,3] T236606 |
[tools] |
| 14:20 |
<jeh> |
create new puppet prefixes for existing (no change in data) and new elasticsearch VMs |
[tools] |
| 04:35 |
<bd808> |
Joined tools-k8s-worker-54 to 2020 Kubernetes cluster |
[tools] |
| 04:34 |
<bd808> |
Joined tools-k8s-worker-53 to 2020 Kubernetes cluster |
[tools] |
| 04:32 |
<bd808> |
Joined tools-k8s-worker-52 to 2020 Kubernetes cluster |
[tools] |
| 04:31 |
<bd808> |
Joined tools-k8s-worker-51 to 2020 Kubernetes cluster |
[tools] |
| 04:28 |
<bd808> |
Joined tools-k8s-worker-50 to 2020 Kubernetes cluster |
[tools] |
| 04:24 |
<bd808> |
Joined tools-k8s-worker-49 to 2020 Kubernetes cluster |
[tools] |
| 04:23 |
<bd808> |
Joined tools-k8s-worker-48 to 2020 Kubernetes cluster |
[tools] |
| 04:21 |
<bd808> |
Joined tools-k8s-worker-47 to 2020 Kubernetes cluster |
[tools] |
| 04:21 |
<bd808> |
Joined tools-k8s-worker-46 to 2020 Kubernetes cluster |
[tools] |
| 04:19 |
<bd808> |
Joined tools-k8s-worker-45 to 2020 Kubernetes cluster |
[tools] |
| 04:14 |
<bd808> |
Joined tools-k8s-worker-44 to 2020 Kubernetes cluster |
[tools] |
| 04:13 |
<bd808> |
Joined tools-k8s-worker-43 to 2020 Kubernetes cluster |
[tools] |
| 04:12 |
<bd808> |
Joined tools-k8s-worker-42 to 2020 Kubernetes cluster |
[tools] |
| 04:10 |
<bd808> |
Joined tools-k8s-worker-41 to 2020 Kubernetes cluster |
[tools] |
| 04:09 |
<bd808> |
Joined tools-k8s-worker-40 to 2020 Kubernetes cluster |
[tools] |
| 04:08 |
<bd808> |
Joined tools-k8s-worker-39 to 2020 Kubernetes cluster |
[tools] |
| 04:07 |
<bd808> |
Joined tools-k8s-worker-38 to 2020 Kubernetes cluster |
[tools] |
| 04:06 |
<bd808> |
Joined tools-k8s-worker-37 to 2020 Kubernetes cluster |
[tools] |
| 03:48 |
<bd808> |
Joined tools-k8s-worker-36 to 2020 Kubernetes cluster |
[tools] |
| 00:50 |
<bstorm_> |
rebuilt all docker images to include webservice 0.64 |
[tools] |
|
2020-02-27
§
|
| 23:27 |
<bstorm_> |
installed toollabs-webservice 0.64 on the bastions |
[tools] |
| 23:23 |
<bstorm_> |
pushed toollabs-webservice version 0.64 to all toolforge repos |
[tools] |
| 21:03 |
<jeh> |
add reindex service account to elasticsearch for data migration T236606 |
[tools] |
| 20:57 |
<bstorm_> |
upgrading toollabs-webservice to stretch-toolsbeta version for jdk8:testing image only |
[tools] |
| 20:19 |
<jeh> |
update elasticsearch VPS security group to allow toolsbeta-elastic7-1 access on tcp 80 T236606 |
[tools] |
| 18:53 |
<bstorm_> |
hard rebooted a rather stuck tools-sgecron-01 |
[tools] |
| 18:20 |
<bd808> |
Building tools-k8s-worker-[36-55] |
[tools] |
| 17:56 |
<bd808> |
Deleted instances tools-worker-10[21-40] |
[tools] |
| 16:14 |
<bd808> |
Decommissioning tools-worker-10[21-40] |
[tools] |
| 16:02 |
<bd808> |
Drained tools-worker-1021 |
[tools] |
| 15:51 |
<bd808> |
Drained tools-worker-1022 |
[tools] |
| 15:44 |
<bd808> |
Drained tools-worker-1023 (there is no tools-worker-1024) |
[tools] |
| 15:39 |
<bd808> |
Drained tools-worker-1025 |
[tools] |
| 15:39 |
<bd808> |
Drained tools-worker-1026 |
[tools] |
| 15:11 |
<bd808> |
Drained tools-worker-1027 |
[tools] |
| 15:09 |
<bd808> |
Drained tools-worker-1028 (there is no tools-worker-1029) |
[tools] |
| 15:07 |
<bd808> |
Drained tools-worker-1030 |
[tools] |
| 15:06 |
<bd808> |
Uncordoned tools-worker-10[16-20]. Was over optimistic about repacking legacy Kubernetes cluster into 15 instances. Will keep 20 for now. |
[tools] |
| 15:00 |
<bd808> |
Drained tools-worker-1031 |
[tools] |
| 14:54 |
<bd808> |
Hard reboot tools-worker-1016. Direct virsh console unresponsive. Stuck in shutdown since 2020-01-22? |
[tools] |
| 14:44 |
<bd808> |
Uncordoned tools-worker-1009.tools.eqiad.wmflabs |
[tools] |
| 14:41 |
<bd808> |
Drained tools-worker-1032 |
[tools] |
| 14:37 |
<bd808> |
Drained tools-worker-1033 |
[tools] |
| 14:35 |
<bd808> |
Drained tools-worker-1034 |
[tools] |