2951-3000 of 7222 results (41ms)
2023-10-18 §
11:32 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-31 from 1.22.17 to 1.23.17 [tools]
11:31 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-30 from 1.22.17 to 1.23.17 [tools]
11:30 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-30 from 1.22.17 to 1.23.17 [tools]
11:29 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-6 from 1.22.17 to 1.23.17 [tools]
11:25 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-6 from 1.22.17 to 1.23.17 [tools]
11:23 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-5 from 1.22.17 to 1.23.17 [tools]
11:16 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-5 from 1.22.17 to 1.23.17 [tools]
11:15 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-4 from 1.22.17 to 1.23.17 [tools]
11:07 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-4 from 1.22.17 to 1.23.17 [tools]
11:04 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.22.17 to 1.23.17 (T298005) [tools]
11:03 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.22.17 to 1.23.17 (T298005) [tools]
2023-10-16 §
09:04 <dcaro> rebooting tools-k8s-worker-45 due to stuck nfs processes [tools]
2023-10-13 §
13:29 <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api [tools]
13:28 <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api [tools]
09:48 <sstefanova@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [tools]
09:48 <sstefanova@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [tools]
09:07 <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) [tools]
09:07 <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors [tools]
08:57 <dcaro> rebooting tools-sgeexec-10-8 as the host is stuck/unreachable [tools]
07:43 <dcaro> rebooting tools-sgeweblight-10-26 as it fails to allocate memory [tools]
2023-10-12 §
15:07 <taavi> reboot tools-k8s-worker-70 [tools]
14:01 <taavi> deploy jobs-cli v15 T348250 [tools]
13:10 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-jobs-framework-cli' version '15' [tools]
13:10 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-jobs-framework-cli' version '15' [tools]
12:21 <dcaro> rebooting sgeexec-10-17 [tools]
12:02 <taavi> also reboot tools-sgeweblight-10-30 [tools]
12:00 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) [tools]
12:00 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors [tools]
11:52 <taavi> reboot tools-sgeweblight-10-22, 28 [tools]
2023-10-11 §
19:47 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers [tools]
17:10 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for all workers [tools]
14:41 <wm-bot2> dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.grid.reboot_workers (exit_code=99) for weblight nodes (T348634) [tools]
14:24 <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.grid.reboot_workers for weblight nodes (T348634) [tools]
14:21 <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [tools]
14:20 <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console [tools]
14:19 <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [tools]
14:19 <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console [tools]
14:16 <dcaro> rebooting tools-sgeweblight-10-16 due to stuck NFS (T348634) [tools]
12:11 <taavi> reboot k8s workers 48, 60, 65, 68, 70, 76 T348634 [tools]
12:04 <taavi> reboot k8s workers 72, 75, 82 T348634 [tools]
12:01 <taavi> reboot tools-sgecron-2 T348634 [tools]
11:49 <taavi> reboot tools-sgeexec-10-19 [tools]
2023-10-10 §
08:30 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [tools]
08:30 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [tools]
2023-10-09 §
10:29 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'python3-toolforge-weld' version '1.4.0' [tools]
10:29 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'python3-toolforge-weld' version '1.4.0' [tools]
08:15 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [tools]
08:15 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [tools]
07:14 <taavi> deploy jobs-framework-cli v14 [tools]
07:13 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-jobs-framework-cli' version '14' [tools]