3151-3200 of 6590 results (34ms)
2022-05-14 §
10:47 <taavi> hard reboot unresponsible tools-sgeexec-0940 [tools]
2022-05-12 §
12:36 <taavi> re-enable CronJobControllerV2 T308205 [tools]
09:28 <taavi> deploy jobs-api update T308204 [tools]
09:15 <wm-bot2> build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:latest from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (e6fa299) (T308204) - cookbook ran by taavi@runko [tools]
2022-05-10 §
15:18 <taavi> depool tools-k8s-worker-42 for experiments [tools]
13:54 <taavi> enable distro-wikimedia unattended upgrades T290494 [tools]
2022-05-06 §
19:46 <bd808> Rebuilt toolforge-perl532-sssd-base & toolforge-perl532-sssd-web to add liblocale-codes-perl (T307812) [tools]
2022-05-05 §
17:28 <taavi> deploy tools-webservice 0.83 T307693 [tools]
2022-05-03 §
08:20 <taavi> redis: start replication from the old cluster to the new one (T278541) [tools]
2022-05-02 §
08:54 <taavi> restart acme-chief.service T307333 [tools]
2022-04-25 §
14:56 <bd808> Rebuilding all docker images to pick up toolforge-webservice v0.82 (T214343) [tools]
14:46 <bd808> Building toolforge-webservice v0.82 [tools]
2022-04-23 §
16:51 <bd808> Built new perl532-sssd/{base,web} images and pushed to registry (T214343) [tools]
2022-04-20 §
16:58 <taavi> reboot toolserver-proxy-01 to free up disk space from stale file handles(?) [tools]
07:51 <wm-bot> build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:latest from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (8f37a04) - cookbook ran by taavi@runko [tools]
2022-04-16 §
18:53 <wm-bot> deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/kubernetes-metrics (2c485e9) - cookbook ran by taavi@runko [tools]
2022-04-12 §
21:32 <bd808> Added komla to Gerrit group 'toollabs-trusted' (T305986) [tools]
21:27 <bd808> Added komla to 'roots' sudoers policy (T305986) [tools]
21:24 <bd808> Add komla as projectadmin (T305986) [tools]
2022-04-10 §
18:43 <taavi> deleted `/tmp/dwl02.out-20210915` on tools-sgebastion-07 (not touched since september, taking up 1.3G of disk space) [tools]
2022-04-09 §
15:30 <taavi> manually prune user.log on tools-prometheus-03 to free up some space on / [tools]
2022-04-08 §
10:44 <arturo> disabled debug mode on the k8s jobs-emailer component [tools]
2022-04-05 §
07:52 <wm-bot> deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (d7d3463) - cookbook ran by arturo@nostromo [tools]
07:44 <wm-bot> build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:latest from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (d7d3463) - cookbook ran by arturo@nostromo [tools]
07:21 <arturo> deploying toolforge-jobs-framework-cli v7 [tools]
2022-04-04 §
17:05 <wm-bot> deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (cbcfc47) - cookbook ran by arturo@nostromo [tools]
16:56 <wm-bot> build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:latest from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (cbcfc47) - cookbook ran by arturo@nostromo [tools]
09:28 <arturo> deployed toolforge-jobs-framework-cli v6 into aptly and installed it on buster bastions [tools]
2022-03-28 §
09:32 <wm-bot> cleaned up grid queue errors on tools-sgegrid-master.tools.eqiad1.wikimedia.cloud (T304816) - cookbook ran by arturo@nostromo [tools]
2022-03-15 §
16:57 <wm-bot> build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-emailer:latest from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer (084ee51) - cookbook ran by arturo@nostromo [tools]
11:24 <arturo> cleared error state on queue continuous@tools-sgeexec-0939.tools.eqiad.wmflabs (a job took a very long time to be scheduled...) [tools]
2022-03-14 §
11:44 <arturo> deploy jobs-framework-emailer 9470a5f339fd5a44c97c69ce97239aef30f5ee41 (T286135) [tools]
10:48 <dcaro> pushed v0.33.2 tekton control and webhook images, and bashA5.1.4 to the local repo (T297090) [tools]
2022-03-10 §
09:42 <arturo> cleaned grid queue error state @ tools-sgewebgrid-generic-0902 [tools]
2022-03-01 §
13:41 <dcaro> rebooting tools-sgeexec-0916 to clear any state (T302702) [tools]
12:11 <dcaro> Cleared error state queues for sgeexec-0916 (T302702) [tools]
10:23 <arturo> tools-sgeeex-0913/0916 are depooled, queue errors. Reboot them and clean errors by hand [tools]
2022-02-28 §
08:02 <taavi> reboot sgeexec-0916 [tools]
07:49 <taavi> depool tools-sgeexec-0916.tools as it is out of disk space on / [tools]
2022-02-17 §
08:23 <taavi> deleted tools-clushmaster-02 [tools]
08:14 <taavi> made tools-puppetmaster-02 its own client to fix `puppet node deactivate` puppetdb access [tools]
2022-02-16 §
00:12 <bd808> Image builds completed. [tools]
2022-02-15 §
23:17 <bd808> Image builds failed in buster php image with an apt error. The error looks transient, so starting builds over. [tools]
23:06 <bd808> Started full rebuild of Toolforge containers to pick up webservice 0.81 and other package updates in tmux session on tools-docker-imagebuilder-01 [tools]
22:58 <bd808> `sudo apt-get update && sudo apt-get install toolforge-webservice` on all bastions to pick up 0.81 [tools]
22:50 <bd808> Built new toollabs-webservice 0.81 [tools]
18:43 <bd808> Enabled puppet on tools-proxy-05 [tools]
18:38 <bd808> Disabled puppet on tools-proxy-05 for manual testing of nginx config changes [tools]
18:21 <taavi> delete tools-package-builder-03 [tools]
11:49 <arturo> invalidate sssd cache in all bastions to debug T301736 [tools]