1-50 of 3834 results (30ms)
2023-04-28 §
15:01 <arturo> force reboot tools-k8s-worker-79, unresponsive [tools]
08:27 <dcaro> rebooting tools-sgeweblight-10-28 (T335336) [tools]
07:20 <dcaro> rebooting tools-sgegrid-shadow due to stale nfs mount [tools]
00:09 <bd808> `kubectl uncordon tools-k8s-worker-67` (T335543) [tools]
00:07 <bd808> Hard reboot tools-k8s-worker-67.tools.eqiad1.wikimedia.cloud via horizon (T335543) [tools]
00:04 <bd808> Rebooting tools-k8s-worker-67.tools.eqiad1.wikimedia.cloud (T335543) [tools]
2023-04-27 §
23:59 <bd808> `kubectl drain --ignore-daemonsets --delete-emptydir-data --force tools-k8s-worker-67` (T335543) [tools]
20:50 <bd808> Started process to rebuild all buster and bullseye based container images again. Prior problem seems to have been stale images in local cache on the build server. [tools]
20:42 <bd808> Container image rebuild failed with GPG errors in buster-sssd base image. Will investigate and attempt to restart once resolved in a local dev environment. [tools]
20:33 <bd808> Started process to rebuild all buster and bullseye based container images per https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes#Building_toolforge_specific_images [tools]
2023-04-18 §
16:46 <dcaro> force-rebooting tools-sgeweblight-10-25/26/27 as they got stuck stopping the grid_exec process [tools]
16:35 <dcaro> rebooting root@tools-sgeweblight-10-27 due to stuck exec daemon not releasing port 6445 [tools]
16:35 <dcaro> rebooting root@tools-sgeweblight-10-25 due to stuck exec daemon not releasing port 6445 [tools]
16:32 <dcaro> rebooting root@tools-sgeweblight-10-26 due to stuck exec daemon not releasing port 6445 [tools]
16:26 <dcaro> rebooting root@tools-sgeexec-10-14 due to stuck exec daemon not releasing port 6445 [tools]
2023-04-17 §
13:10 <dcaro> rebooting tools-sgegrid-master node (T334847) [tools]
02:43 <legoktm> manual restart of apache2 on toolserver-proxy-1 to completely pick up renewed TLS cert (alert was flapping) [tools]
2023-04-11 §
16:11 <wm-bot2> deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (b65439b) - cookbook ran by arturo@nostromo [tools]
15:46 <arturo> upload toolforge-jobs-framework-cli v11 to aptly [tools]
14:17 <wm-bot2> deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller.git (d878e49) (T324834) - cookbook ran by dcaro@vulcanus [tools]
13:19 <wm-bot2> build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:c6c693c from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (c6c693c) - cookbook ran by arturo@nostromo [tools]
12:09 <wm-bot2> build & push docker image docker-registry.tools.wmflabs.org/volume-admission:40bd3b3 from https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller (40bd3b3) - cookbook ran by dcaro@vulcanus [tools]
10:34 <wm-bot2> deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx (9aed7e5) - cookbook ran by taavi@runko [tools]
09:15 <wm-bot2> deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/calico (c6a3e29) (T329677) - cookbook ran by taavi@runko [tools]
08:45 <wm-bot2> Adding a new k8s worker node - cookbook ran by taavi@runko [tools]
2023-04-10 §
10:46 <taavi> patch existing PSP roles to use policy/v1beta1 T331619 [tools]
09:16 <arturo> upgrading k8s cluster to 1.22 (T286856) [tools]
2023-04-07 §
14:34 <wm-bot2> drained, depooled and removed k8s control node tools-k8s-control-3 (T333929) - cookbook ran by taavi@runko [tools]
14:30 <wm-bot2> removed instance tools-k8s-control-2 - cookbook ran by taavi@runko [tools]
2023-04-05 §
15:16 <wm-bot2> deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (5ea5992) - cookbook ran by taavi@runko [tools]
15:10 <wm-bot2> build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:3569803 from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (3569803) - cookbook ran by taavi@runko [tools]
14:56 <wm-bot2> Added a new k8s worker tools-k8s-worker-88.tools.eqiad1.wikimedia.cloud to the cluster (T333972) - cookbook ran by taavi@runko [tools]
14:42 <wm-bot2> Adding a new k8s worker node (T333972) - cookbook ran by taavi@runko [tools]
14:42 <wm-bot2> Added a new k8s worker tools-k8s-worker-87.tools.eqiad1.wikimedia.cloud to the cluster (T333972) - cookbook ran by taavi@runko [tools]
14:28 <wm-bot2> Adding a new k8s worker node (T333972) - cookbook ran by taavi@runko [tools]
14:28 <wm-bot2> Added a new k8s worker tools-k8s-worker-86.tools.eqiad1.wikimedia.cloud to the cluster (T333972) - cookbook ran by taavi@runko [tools]
14:15 <wm-bot2> Adding a new k8s worker node (T333972) - cookbook ran by taavi@runko [tools]
14:15 <wm-bot2> Added a new k8s worker tools-k8s-worker-85.tools.eqiad1.wikimedia.cloud to the cluster (T333972) - cookbook ran by taavi@runko [tools]
14:01 <wm-bot2> Adding a new k8s worker node (T333972) - cookbook ran by taavi@runko [tools]
14:01 <wm-bot2> Added a new k8s worker tools-k8s-worker-84.tools.eqiad1.wikimedia.cloud to the cluster (T333972) - cookbook ran by taavi@runko [tools]
13:47 <wm-bot2> Adding a new k8s worker node (T333972) - cookbook ran by taavi@runko [tools]
13:47 <wm-bot2> Added a new k8s worker tools-k8s-worker-83.tools.eqiad1.wikimedia.cloud to the cluster (T333972) - cookbook ran by taavi@runko [tools]
13:34 <wm-bot2> Adding a new k8s worker node (T333972) - cookbook ran by taavi@runko [tools]
13:33 <wm-bot2> removed instance tools-k8s-worker-83 - cookbook ran by taavi@runko [tools]
13:15 <wm-bot2> Adding a new k8s worker node (T333972) - cookbook ran by taavi@runko [tools]
13:06 <wm-bot2> removing grid node tools-sgeweblight-10-31.tools.eqiad1.wikimedia.cloud (T333972) - cookbook ran by taavi@runko [tools]
13:02 <wm-bot2> removing grid node tools-sgeweblight-10-29.tools.eqiad1.wikimedia.cloud (T333972) - cookbook ran by taavi@runko [tools]
13:00 <wm-bot2> removing grid node tools-sgeexec-10-9.tools.eqiad1.wikimedia.cloud (T333972) - cookbook ran by taavi@runko [tools]
12:58 <wm-bot2> removing grid node tools-sgeweblight-10-15.tools.eqiad1.wikimedia.cloud (T333972) - cookbook ran by taavi@runko [tools]
12:54 <wm-bot2> removing grid node tools-sgeexec-10-7.tools.eqiad1.wikimedia.cloud (T333972) - cookbook ran by taavi@runko [tools]