1-50 of 3427 results (20ms)
2022-04-20 §
16:58 <taavi> reboot toolserver-proxy-01 to free up disk space from stale file handles(?) [tools]
07:51 <wm-bot> build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:latest from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (8f37a04) - cookbook ran by taavi@runko [tools]
2022-04-16 §
18:53 <wm-bot> deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/kubernetes-metrics (2c485e9) - cookbook ran by taavi@runko [tools]
2022-04-12 §
21:32 <bd808> Added komla to Gerrit group 'toollabs-trusted' (T305986) [tools]
21:27 <bd808> Added komla to 'roots' sudoers policy (T305986) [tools]
21:24 <bd808> Add komla as projectadmin (T305986) [tools]
2022-04-10 §
18:43 <taavi> deleted `/tmp/dwl02.out-20210915` on tools-sgebastion-07 (not touched since september, taking up 1.3G of disk space) [tools]
2022-04-09 §
15:30 <taavi> manually prune user.log on tools-prometheus-03 to free up some space on / [tools]
2022-04-08 §
10:44 <arturo> disabled debug mode on the k8s jobs-emailer component [tools]
2022-04-05 §
07:52 <wm-bot> deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (d7d3463) - cookbook ran by arturo@nostromo [tools]
07:44 <wm-bot> build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:latest from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (d7d3463) - cookbook ran by arturo@nostromo [tools]
07:21 <arturo> deploying toolforge-jobs-framework-cli v7 [tools]
2022-04-04 §
17:05 <wm-bot> deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (cbcfc47) - cookbook ran by arturo@nostromo [tools]
16:56 <wm-bot> build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:latest from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (cbcfc47) - cookbook ran by arturo@nostromo [tools]
09:28 <arturo> deployed toolforge-jobs-framework-cli v6 into aptly and installed it on buster bastions [tools]
2022-03-28 §
09:32 <wm-bot> cleaned up grid queue errors on tools-sgegrid-master.tools.eqiad1.wikimedia.cloud (T304816) - cookbook ran by arturo@nostromo [tools]
2022-03-15 §
16:57 <wm-bot> build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-emailer:latest from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer (084ee51) - cookbook ran by arturo@nostromo [tools]
11:24 <arturo> cleared error state on queue continuous@tools-sgeexec-0939.tools.eqiad.wmflabs (a job took a very long time to be scheduled...) [tools]
2022-03-14 §
11:44 <arturo> deploy jobs-framework-emailer 9470a5f339fd5a44c97c69ce97239aef30f5ee41 (T286135) [tools]
10:48 <dcaro> pushed v0.33.2 tekton control and webhook images, and bashA5.1.4 to the local repo (T297090) [tools]
2022-03-10 §
09:42 <arturo> cleaned grid queue error state @ tools-sgewebgrid-generic-0902 [tools]
2022-03-01 §
13:41 <dcaro> rebooting tools-sgeexec-0916 to clear any state (T302702) [tools]
12:11 <dcaro> Cleared error state queues for sgeexec-0916 (T302702) [tools]
10:23 <arturo> tools-sgeeex-0913/0916 are depooled, queue errors. Reboot them and clean errors by hand [tools]
2022-02-28 §
08:02 <taavi> reboot sgeexec-0916 [tools]
07:49 <taavi> depool tools-sgeexec-0916.tools as it is out of disk space on / [tools]
2022-02-17 §
08:23 <taavi> deleted tools-clushmaster-02 [tools]
08:14 <taavi> made tools-puppetmaster-02 its own client to fix `puppet node deactivate` puppetdb access [tools]
2022-02-16 §
00:12 <bd808> Image builds completed. [tools]
2022-02-15 §
23:17 <bd808> Image builds failed in buster php image with an apt error. The error looks transient, so starting builds over. [tools]
23:06 <bd808> Started full rebuild of Toolforge containers to pick up webservice 0.81 and other package updates in tmux session on tools-docker-imagebuilder-01 [tools]
22:58 <bd808> `sudo apt-get update && sudo apt-get install toolforge-webservice` on all bastions to pick up 0.81 [tools]
22:50 <bd808> Built new toollabs-webservice 0.81 [tools]
18:43 <bd808> Enabled puppet on tools-proxy-05 [tools]
18:38 <bd808> Disabled puppet on tools-proxy-05 for manual testing of nginx config changes [tools]
18:21 <taavi> delete tools-package-builder-03 [tools]
11:49 <arturo> invalidate sssd cache in all bastions to debug T301736 [tools]
11:16 <arturo> purge debian package `unscd` on tools-sgebastion-10/11 for T301736 [tools]
11:14 <arturo> reboot tools-sgebastion-10 for T301736 [tools]
2022-02-10 §
15:07 <taavi> shutdown tools-clushmaster-02 T298191 [tools]
13:25 <wm-bot> trying to join node tools-sgewebgen-10-2 to the grid cluster in tools. - cookbook ran by arturo@nostromo [tools]
13:24 <wm-bot> trying to join node tools-sgewebgen-10-1 to the grid cluster in tools. - cookbook ran by arturo@nostromo [tools]
13:07 <wm-bot> trying to join node tools-sgeweblight-10-5 to the grid cluster in tools. - cookbook ran by arturo@nostromo [tools]
13:06 <wm-bot> trying to join node tools-sgeweblight-10-4 to the grid cluster in tools. - cookbook ran by arturo@nostromo [tools]
13:05 <wm-bot> trying to join node tools-sgeweblight-10-3 to the grid cluster in tools. - cookbook ran by arturo@nostromo [tools]
13:03 <wm-bot> trying to join node tools-sgeweblight-10-2 to the grid cluster in tools. - cookbook ran by arturo@nostromo [tools]
12:54 <wm-bot> trying to join node tools-sgeweblight-10-1.tools.eqiad1.wikimedia.cloud to the grid cluster in tools. - cookbook ran by arturo@nostromo [tools]
08:45 <taavi> set `profile::base::manage_ssh_keys: true` globally T214427 [tools]
08:16 <taavi> enable puppetdb and re-enable puppet with puppetdb ssh key management disabled (profile::base::manage_ssh_keys: false) - T214427 [tools]
08:06 <taavi> disable puppet globally for enabling puppetdb T214427 [tools]