2351-2400 of 7537 results (38ms)
2024-03-18 §
11:17 <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.23.17 to 1.24.17 (T359638) [tools]
11:13 <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.23.17 to 1.24.17 (T359638) [tools]
11:08 <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.23.17 to 1.24.17 (T359638) [tools]
11:01 <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics [tools]
11:00 <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics [tools]
11:00 <aborrero@cloudcumin1001> END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-api [tools]
11:00 <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [tools]
11:00 <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.23.17 to 1.24.17 (T359638) [tools]
10:53 <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.23.17 to 1.24.17 (T359638) [tools]
10:53 <aborrero@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-7 from 1.23.17 to 1.24.17 (T359638) [tools]
10:53 <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-7 from 1.23.17 to 1.24.17 (T359638) [tools]
10:47 <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.23.17 to 1.24.17 (T307651) [tools]
10:46 <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.23.17 to 1.24.17 (T307651) [tools]
10:04 <taavi@cloudcumin1001> END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-bastion-12.tools.eqiad1.wikimedia.cloud [tools]
10:03 <taavi@cloudcumin1001> START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-12.tools.eqiad1.wikimedia.cloud [tools]
09:27 <taavi> deleted shutdown grid engine VMs T314664 [tools]
2024-03-15 §
10:50 <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [tools]
10:50 <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [tools]
2024-03-14 §
17:26 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'misctools' version '1.48' [tools]
17:26 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'misctools' version '1.48' [tools]
15:16 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-imagebuilder-01 [tools]
15:16 <taavi@cloudcumin1001> START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01 [tools]
15:11 <taavi@cloudcumin1001> END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01 [tools]
15:11 <taavi@cloudcumin1001> START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01 [tools]
15:10 <taavi@cloudcumin1001> END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01 [tools]
15:09 <taavi@cloudcumin1001> START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01 [tools]
11:02 <taavi> stop grid related VMs T314664 [tools]
11:01 <taavi> disable grid access for remaining tools still running on the grid T314664 [tools]
2024-03-13 §
19:21 <andrewbogott> shutting down old puppet infra: tools-puppetmaster-02 and tools-puppetdb-1. These can be deleted in a week or two presuming everything remains stable. [tools]
2024-03-12 §
12:38 <taavi> hard reboot tools-prometheus-6 [tools]
11:50 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [tools]
11:50 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [tools]
2024-03-11 §
16:46 <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics [tools]
16:46 <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics [tools]
13:20 <arturo> cached registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.6.0 as docker-registry.tools.wmflabs.org/kube-state-metrics:v2.6.0 in the docker registry for T359798 [tools]
2024-03-09 §
12:48 <taavi> hard reboot tools-sgebastion-10 due to stuck NFS procs [tools]
2024-03-08 §
12:02 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [tools]
12:02 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [tools]
2024-03-07 §
14:33 <wmbot~dcaro@urcuchillay> END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [tools]
14:32 <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [tools]
13:42 <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [tools]
13:41 <aborrero@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [tools]
2024-03-06 §
10:48 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-32 [tools]
10:47 <taavi@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.remove_grid_node (exit_code=1) for tools-sgeweblight-10-17, tools-sgeweblight-10-32 [tools]
10:47 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-32 [tools]
10:34 <taavi> rebuilding all docker images for https://gerrit.wikimedia.org/r/c/operations/docker-images/toollabs-images/+/1005952 (T293552) + normal package updates [tools]
09:43 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) [tools]
09:43 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors [tools]
09:42 <taavi> reboot tools-sgeexec-10-20, -21, -23, sgeweblight-10-32 due to stuck nfs procs [tools]
2024-03-05 §
16:12 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud [tools]