2001-2050 of 6424 results (41ms)
2023-11-17 §
15:49 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api [tools]
2023-11-16 §
21:08 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers [tools]
19:54 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for all workers [tools]
13:47 <taavi> reboot tools-sgecron-2 with very high load average [tools]
2023-11-14 §
19:03 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [tools]
19:03 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [tools]
10:11 <taavi> reboot unresponsive tools-sgeexec-10-22 [tools]
2023-11-13 §
22:21 <taavi> reboot! tools-sgewebgen-10-3, tools-sgeweblight-10-21, tools-sgeweblight-10-32, tools-sgeexec-10-16 due to high load average and/or stuck jobs [tools]
16:37 <taavi> drain tools-k8s-worker-84 tools-k8s-worker-85 [tools]
2023-11-09 §
11:49 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [tools]
11:49 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [tools]
2023-11-07 §
11:45 <taavi> reboot tools-sgeexec-10-8 which had high load average [tools]
2023-11-02 §
13:13 <taavi> wiping data directory from tools-prometheus-7 so we have least one working server T350227 [tools]
2023-11-01 §
14:19 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics [tools]
14:19 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics [tools]
09:06 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) [tools]
09:06 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors [tools]
08:47 <taavi> restart puppetdb [tools]
2023-10-30 §
14:03 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics [tools]
14:02 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics [tools]
2023-10-29 §
17:46 <andrewbogott> running SET GLOBAL read_only=OFF; for mariadb on tools-db-1.tools.eqiad1.wikimedia.cloud [tools]
17:37 <andrewbogott> rebooting tools-db-1.tools.eqiad1.wikimedia.cloud to recover from the oom-killer firing [tools]
2023-10-26 §
08:29 <taavi> root@tools-sgeweblight-10-21:~# sudo dpkg --configure -a [tools]
08:18 <taavi> restart sssd on tools-nfs-2 [tools]
2023-10-25 §
09:08 <blancadesal> harbor up again and upgraded from 2.5 to 2.9 (T346241) [tools]
08:31 <blancadesal> taking harbor down for upgrade (T346241) [tools]
2023-10-24 §
16:02 <taavi> reboot tools-sgeweblight-10-28 [tools]
09:49 <taavi> reboot tools-sgebastion-11 due to high load [tools]
09:35 <taavi> make ToolsDBWritableState alert paging, match icinga check removed in https://gerrit.wikimedia.org/r/c/operations/puppet/+/956071 [tools]
2023-10-23 §
15:40 <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [tools]
15:39 <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [tools]
14:18 <dcaro> release toolforge-builds-cli 0.0.4 [tools]
08:22 <taavi> reboot tools-sgeweblight-10-14, 24 T349425 [tools]
2023-10-19 §
12:48 <taavi> flush queued webgrid jobs that had been waiting in the queue since the nfs issues last week [tools]
2023-10-18 §
12:21 <sstefanova@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [tools]
12:21 <sstefanova@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [tools]
12:05 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-70 from 1.22.17 to 1.23.17 [tools]
12:03 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-88 from 1.22.17 to 1.23.17 [tools]
12:03 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-70 from 1.22.17 to 1.23.17 [tools]
12:03 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-69 from 1.22.17 to 1.23.17 [tools]
12:03 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-50 from 1.22.17 to 1.23.17 [tools]
12:02 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-88 from 1.22.17 to 1.23.17 [tools]
12:02 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-87 from 1.22.17 to 1.23.17 [tools]
12:02 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-69 from 1.22.17 to 1.23.17 [tools]
12:02 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-68 from 1.22.17 to 1.23.17 [tools]
12:01 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-50 from 1.22.17 to 1.23.17 [tools]
12:01 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-49 from 1.22.17 to 1.23.17 [tools]
12:01 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-87 from 1.22.17 to 1.23.17 [tools]
12:00 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-86 from 1.22.17 to 1.23.17 [tools]
12:00 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-68 from 1.22.17 to 1.23.17 [tools]