1301-1350 of 5145 results (39ms)
2023-05-08 §
09:11 <arturo> force-reboot tools-sgeexec-10-13 (reported as down by the monitoring, no SSH) [tools]
2023-05-07 §
16:06 <taavi> remove inbound 25/tcp rule from the toolserver legacy server T136225 [tools]
2023-05-05 §
22:21 <bd808> Added "RepoLookoutBot" to hiera key "dynamicproxy::blocked_user_agent_regex" to stop unnecessary scans by https://www.repo-lookout.org/ [tools]
22:20 <bd808> Added [tools]
11:30 <wm-bot2> build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:811164e from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (811164e) - cookbook ran by taavi@runko [tools]
09:13 <dcaro> rebooted tools-sgeexec-10-16 as it was stuck (T335009) [tools]
2023-05-04 §
15:15 <wm-bot2> removed instance tools-k8s-etcd-15 - cookbook ran by andrew@bullseye [tools]
14:13 <wm-bot2> removed instance tools-k8s-etcd-14 - cookbook ran by andrew@bullseye [tools]
2023-05-03 §
12:41 <wm-bot2> removed instance tools-k8s-etcd-13 - cookbook ran by andrew@bullseye [tools]
2023-05-02 §
00:29 <wm-bot2> deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller (7199a9e) - cookbook ran by raymond@ubuntu [tools]
2023-05-01 §
23:17 <wm-bot2> build & push docker image docker-registry.tools.wmflabs.org/toolforge-buildpack-admission-controller:3b3803f from https://github.com/toolforge/buildpack-admission-controller (3b3803f) - cookbook ran by raymond@ubuntu [tools]
2023-04-28 §
15:01 <arturo> force reboot tools-k8s-worker-79, unresponsive [tools]
08:27 <dcaro> rebooting tools-sgeweblight-10-28 (T335336) [tools]
07:20 <dcaro> rebooting tools-sgegrid-shadow due to stale nfs mount [tools]
00:09 <bd808> `kubectl uncordon tools-k8s-worker-67` (T335543) [tools]
00:07 <bd808> Hard reboot tools-k8s-worker-67.tools.eqiad1.wikimedia.cloud via horizon (T335543) [tools]
00:04 <bd808> Rebooting tools-k8s-worker-67.tools.eqiad1.wikimedia.cloud (T335543) [tools]
2023-04-27 §
23:59 <bd808> `kubectl drain --ignore-daemonsets --delete-emptydir-data --force tools-k8s-worker-67` (T335543) [tools]
20:50 <bd808> Started process to rebuild all buster and bullseye based container images again. Prior problem seems to have been stale images in local cache on the build server. [tools]
20:42 <bd808> Container image rebuild failed with GPG errors in buster-sssd base image. Will investigate and attempt to restart once resolved in a local dev environment. [tools]
20:33 <bd808> Started process to rebuild all buster and bullseye based container images per https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes#Building_toolforge_specific_images [tools]
2023-04-18 §
16:46 <dcaro> force-rebooting tools-sgeweblight-10-25/26/27 as they got stuck stopping the grid_exec process [tools]
16:35 <dcaro> rebooting root@tools-sgeweblight-10-27 due to stuck exec daemon not releasing port 6445 [tools]
16:35 <dcaro> rebooting root@tools-sgeweblight-10-25 due to stuck exec daemon not releasing port 6445 [tools]
16:32 <dcaro> rebooting root@tools-sgeweblight-10-26 due to stuck exec daemon not releasing port 6445 [tools]
16:26 <dcaro> rebooting root@tools-sgeexec-10-14 due to stuck exec daemon not releasing port 6445 [tools]
2023-04-17 §
13:10 <dcaro> rebooting tools-sgegrid-master node (T334847) [tools]
02:43 <legoktm> manual restart of apache2 on toolserver-proxy-1 to completely pick up renewed TLS cert (alert was flapping) [tools]
2023-04-11 §
16:11 <wm-bot2> deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (b65439b) - cookbook ran by arturo@nostromo [tools]
15:46 <arturo> upload toolforge-jobs-framework-cli v11 to aptly [tools]
14:17 <wm-bot2> deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller.git (d878e49) (T324834) - cookbook ran by dcaro@vulcanus [tools]
13:19 <wm-bot2> build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:c6c693c from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (c6c693c) - cookbook ran by arturo@nostromo [tools]
12:09 <wm-bot2> build & push docker image docker-registry.tools.wmflabs.org/volume-admission:40bd3b3 from https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller (40bd3b3) - cookbook ran by dcaro@vulcanus [tools]
10:34 <wm-bot2> deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx (9aed7e5) - cookbook ran by taavi@runko [tools]
09:15 <wm-bot2> deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/calico (c6a3e29) (T329677) - cookbook ran by taavi@runko [tools]
08:45 <wm-bot2> Adding a new k8s worker node - cookbook ran by taavi@runko [tools]
2023-04-10 §
10:46 <taavi> patch existing PSP roles to use policy/v1beta1 T331619 [tools]
09:16 <arturo> upgrading k8s cluster to 1.22 (T286856) [tools]
2023-04-07 §
14:34 <wm-bot2> drained, depooled and removed k8s control node tools-k8s-control-3 (T333929) - cookbook ran by taavi@runko [tools]
14:30 <wm-bot2> removed instance tools-k8s-control-2 - cookbook ran by taavi@runko [tools]
2023-04-05 §
15:16 <wm-bot2> deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (5ea5992) - cookbook ran by taavi@runko [tools]
15:10 <wm-bot2> build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:3569803 from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (3569803) - cookbook ran by taavi@runko [tools]
14:56 <wm-bot2> Added a new k8s worker tools-k8s-worker-88.tools.eqiad1.wikimedia.cloud to the cluster (T333972) - cookbook ran by taavi@runko [tools]
14:42 <wm-bot2> Adding a new k8s worker node (T333972) - cookbook ran by taavi@runko [tools]
14:42 <wm-bot2> Added a new k8s worker tools-k8s-worker-87.tools.eqiad1.wikimedia.cloud to the cluster (T333972) - cookbook ran by taavi@runko [tools]
14:28 <wm-bot2> Adding a new k8s worker node (T333972) - cookbook ran by taavi@runko [tools]
14:28 <wm-bot2> Added a new k8s worker tools-k8s-worker-86.tools.eqiad1.wikimedia.cloud to the cluster (T333972) - cookbook ran by taavi@runko [tools]
14:15 <wm-bot2> Adding a new k8s worker node (T333972) - cookbook ran by taavi@runko [tools]
14:15 <wm-bot2> Added a new k8s worker tools-k8s-worker-85.tools.eqiad1.wikimedia.cloud to the cluster (T333972) - cookbook ran by taavi@runko [tools]
14:01 <wm-bot2> Adding a new k8s worker node (T333972) - cookbook ran by taavi@runko [tools]