651-700 of 4054 results (27ms)
2022-03-01 §
10:23 <arturo> tools-sgeeex-0913/0916 are depooled, queue errors. Reboot them and clean errors by hand [tools]
2022-02-28 §
08:02 <taavi> reboot sgeexec-0916 [tools]
07:49 <taavi> depool tools-sgeexec-0916.tools as it is out of disk space on / [tools]
2022-02-17 §
08:23 <taavi> deleted tools-clushmaster-02 [tools]
08:14 <taavi> made tools-puppetmaster-02 its own client to fix `puppet node deactivate` puppetdb access [tools]
2022-02-16 §
00:12 <bd808> Image builds completed. [tools]
2022-02-15 §
23:17 <bd808> Image builds failed in buster php image with an apt error. The error looks transient, so starting builds over. [tools]
23:06 <bd808> Started full rebuild of Toolforge containers to pick up webservice 0.81 and other package updates in tmux session on tools-docker-imagebuilder-01 [tools]
22:58 <bd808> `sudo apt-get update && sudo apt-get install toolforge-webservice` on all bastions to pick up 0.81 [tools]
22:50 <bd808> Built new toollabs-webservice 0.81 [tools]
18:43 <bd808> Enabled puppet on tools-proxy-05 [tools]
18:38 <bd808> Disabled puppet on tools-proxy-05 for manual testing of nginx config changes [tools]
18:21 <taavi> delete tools-package-builder-03 [tools]
11:49 <arturo> invalidate sssd cache in all bastions to debug T301736 [tools]
11:16 <arturo> purge debian package `unscd` on tools-sgebastion-10/11 for T301736 [tools]
11:14 <arturo> reboot tools-sgebastion-10 for T301736 [tools]
2022-02-10 §
15:07 <taavi> shutdown tools-clushmaster-02 T298191 [tools]
13:25 <wm-bot> trying to join node tools-sgewebgen-10-2 to the grid cluster in tools. - cookbook ran by arturo@nostromo [tools]
13:24 <wm-bot> trying to join node tools-sgewebgen-10-1 to the grid cluster in tools. - cookbook ran by arturo@nostromo [tools]
13:07 <wm-bot> trying to join node tools-sgeweblight-10-5 to the grid cluster in tools. - cookbook ran by arturo@nostromo [tools]
13:06 <wm-bot> trying to join node tools-sgeweblight-10-4 to the grid cluster in tools. - cookbook ran by arturo@nostromo [tools]
13:05 <wm-bot> trying to join node tools-sgeweblight-10-3 to the grid cluster in tools. - cookbook ran by arturo@nostromo [tools]
13:03 <wm-bot> trying to join node tools-sgeweblight-10-2 to the grid cluster in tools. - cookbook ran by arturo@nostromo [tools]
12:54 <wm-bot> trying to join node tools-sgeweblight-10-1.tools.eqiad1.wikimedia.cloud to the grid cluster in tools. - cookbook ran by arturo@nostromo [tools]
08:45 <taavi> set `profile::base::manage_ssh_keys: true` globally T214427 [tools]
08:16 <taavi> enable puppetdb and re-enable puppet with puppetdb ssh key management disabled (profile::base::manage_ssh_keys: false) - T214427 [tools]
08:06 <taavi> disable puppet globally for enabling puppetdb T214427 [tools]
2022-02-09 §
19:29 <taavi> installed tools-puppetdb-1, not configured on puppetmaster side yet T214427 [tools]
18:56 <wm-bot> pooled 10 grid nodes tools-sgeweblight-10-[1-5],tools-sgewebgen-10-[1,2],tools-sgeexec-10-[1-10] (T277653) - cookbook ran by arturo@nostromo [tools]
18:30 <wm-bot> pooled 9 grid nodes tools-sgeexec-10-[2-10],tools-sgewebgen-[3,15] - cookbook ran by arturo@nostromo [tools]
18:25 <arturo> ignore last message [tools]
18:24 <wm-bot> pooled 9 grid nodes tools-sgeexec-10-[2-10],tools-sgewebgen-[3,15] - cookbook ran by arturo@nostromo [tools]
14:04 <taavi> created tools-cumin-1/toolsbeta-cumin-1 T298191 [tools]
2022-02-07 §
17:37 <taavi> generated authdns_acmechief ssh key and stored password in a text file in local labs/private repository (T288406) [tools]
12:52 <taavi> updated maintain-kubeusers for T301081 [tools]
2022-02-04 §
22:33 <taavi> `root@tools-sgebastion-10:/data/project/ru_monuments/.kube# mv config old_config` # experimenting with T301015 [tools]
21:36 <taavi> clear error state from some webgrid nodes [tools]
2022-02-03 §
09:06 <taavi> run `sudo apt-get clean` on login-buster/dev-buster to clean up disk space [tools]
08:01 <taavi> restart acme-chief to force renewal of toolserver.org certificate [tools]
2022-01-30 §
14:41 <taavi> created a neutron port with ip 172.16.2.46 for a service ip for toolforge redis automatic failover T278541 [tools]
14:22 <taavi> creating a cluster of 3 bullseye redis hosts for T278541 [tools]
2022-01-26 §
18:33 <wm-bot> depooled grid node tools-sgeexec-10-10 - cookbook ran by arturo@nostromo [tools]
18:33 <wm-bot> depooled grid node tools-sgeexec-10-9 - cookbook ran by arturo@nostromo [tools]
18:33 <wm-bot> depooled grid node tools-sgeexec-10-8 - cookbook ran by arturo@nostromo [tools]
18:32 <wm-bot> depooled grid node tools-sgeexec-10-7 - cookbook ran by arturo@nostromo [tools]
18:32 <wm-bot> depooled grid node tools-sgeexec-10-6 - cookbook ran by arturo@nostromo [tools]
18:31 <wm-bot> depooled grid node tools-sgeexec-10-5 - cookbook ran by arturo@nostromo [tools]
18:30 <wm-bot> depooled grid node tools-sgeexec-10-4 - cookbook ran by arturo@nostromo [tools]
18:28 <wm-bot> depooled grid node tools-sgeexec-10-3 - cookbook ran by arturo@nostromo [tools]
18:27 <wm-bot> depooled grid node tools-sgeexec-10-2 - cookbook ran by arturo@nostromo [tools]