51-100 of 3627 results (14ms)
2022-10-13 §
15:10 <arturo> restart jobs-emailer pod [tools]
2022-10-12 §
23:25 <bd808> Rebuilding all Toolforge docker images (T278436, T311466, T293552) [tools]
20:43 <bd808> Rebuilding all Toolforge docker images to pick up bug and security fix packages. Third try seems to be working. (T316554) [tools]
20:31 <bd808> Rebuilding all Toolforge docker images to pick up bug and security fix packages after fixing bug in building the bullseye base image. (T316554) [tools]
16:26 <dcaro> deploy the latest registry admission webhook, now for real (image tag 07bc7db) [tools]
12:48 <dcaro> deploy the latest registry admission webhook (image tag 07bc7db) [tools]
09:26 <wm-bot2> cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by dcaro@vulcanus [tools]
09:18 <wm-bot2> cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by dcaro@vulcanus [tools]
2022-10-11 §
13:52 <wm-bot2> build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:8574c36 from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (8574c36) - cookbook ran by taavi@runko [tools]
2022-10-10 §
19:30 <taavi> rebooting all k8s worker nodes to clean up labstore1006/7 remains [tools]
16:51 <taavi> clean up labstore1006/7 mounts from k8s control nodes T320425 [tools]
11:35 <arturo> aborrero@tools-k8s-control-1:~$ sudo -i kubectl -n jobs-emailer rollout restart deployment/jobs-emailer (T317998) [tools]
08:44 <wm-bot2> deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (afa90ed) (T320284) - cookbook ran by taavi@runko [tools]
08:39 <wm-bot2> build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:latest from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (afa90ed) - cookbook ran by taavi@runko [tools]
2022-10-09 §
17:29 <taavi> kill 10 idle tmux sessions of user 'hoi' on tools-sgebastion-10 T320352 [tools]
2022-10-07 §
13:02 <taavi> taavi@cloudcontrol1005 ~ $ sudo mark_tool --disable oncall # T320240 [tools]
2022-10-06 §
00:39 <bd808> Image rebuild failing with debian apt repo signature issue. Will investigate tomorrow. (T316554) [tools]
00:36 <bd808> Rebuilding all Toolforge docker images to pick up bug and security fix packages. (T316554) [tools]
00:04 <bd808> Building new php74-sssd-base & web images (T310435) [tools]
2022-10-03 §
14:36 <wm-bot2> build & push docker image docker-registry.tools.wmflabs.org/volume-admission:latest from https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller (8da432b) - cookbook ran by taavi@runko [tools]
2022-09-28 §
21:23 <lucaswerkmeister> on tools-sgebastion-10: run-puppet-agent # T318858 [tools]
21:22 <lucaswerkmeister> on tools-sgebastion-10: apt remove emacs-common emacs-bin-common # fix package conflict, T318858 [tools]
21:15 <lucaswerkmeister> added root SSH key for myself, manually ran puppet on tools-sgebastion-10 to apply it (seemingly successfully) [tools]
2022-09-22 §
12:30 <taavi> add TheresNoTime to the 'toollabs-trusted' gerrit group T317438 [tools]
12:27 <taavi> add TheresNoTime as a project admin and to the roots sudo policy T317438 [tools]
2022-09-10 §
07:38 <wm-bot2> removing instance tools-prometheus-03 - cookbook ran by taavi@runko [tools]
2022-09-07 §
10:22 <dcaro> Pushing the new toolforge builder image based on the new 0.8 buildpacks (T316854) [tools]
2022-09-06 §
08:06 <dcaro_away> Published new toolforge-bullseye0-run and toolforge-bullseye0-build images for the toolforge buildpack builder (T316854) [tools]
2022-08-25 §
10:40 <taavi> tagged new version of the python39-web container with a shell implementation of webservice-runner T293552 [tools]
2022-08-24 §
12:20 <wm-bot2> deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx (eba66bc) - cookbook ran by taavi@runko [tools]
12:20 <taavi> upgrading ingress-nginx to v1.3 [tools]
2022-08-20 §
07:44 <dcaro_away> all k8s nodes ready now \o/ (T315718) [tools]
07:43 <dcaro_away> rebooted tools-k8s-control-2, seemed stuck trying to wait for tools home (nfs?), after reboot came back up (T315718) [tools]
07:41 <dcaro_away> cloudvirt1023 down took out 3 workers, 1 control, and a grid exec and a weblight, they are taking long to restart, looking (T315718) [tools]
2022-08-18 §
14:45 <andrewbogott> adding lucaswerkmeister as projectadmin (T314527) [tools]
14:43 <andrewbogott> removing some inactive projectadmins: rush, petrb, mdipietro, jeh, krenair [tools]
2022-08-17 §
16:34 <taavi> kubectl sudo delete cm -n tool-wdml maintain-kubeusers # T315459 [tools]
08:30 <taavi> failing the grid from the shadow back to the master, some disruption expected [tools]
2022-08-16 §
17:28 <taavi> fail over docker-registry, tools-docker-registry-06->docker-registry-05 [tools]
2022-08-11 §
16:57 <wm-bot2> cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by taavi@runko [tools]
16:55 <taavi> restart puppetdb on tools-puppetdb-1, crashed during the ceph issues [tools]
2022-08-05 §
15:08 <wm-bot2> removing grid node tools-sgewebgen-10-1.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
15:05 <wm-bot2> removing grid node tools-sgeexec-10-12.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
15:00 <wm-bot2> created node tools-sgewebgen-10-3.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko [tools]
2022-08-03 §
15:51 <dhinus> recreated jobs-api pods to pick up new ConfigMap [tools]
15:02 <wm-bot2> deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (c47ac41) - cookbook ran by fran@MacBook-Pro.station [tools]
2022-07-20 §
19:31 <taavi> reboot toolserver-proxy-01 to free up disk space probably held by stale file handles [tools]
08:06 <wm-bot2> removing grid node tools-sgeexec-10-6.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]
2022-07-19 §
17:53 <wm-bot2> created node tools-sgeexec-10-21.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko [tools]
17:00 <wm-bot2> removing grid node tools-sgeexec-10-3.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko [tools]