1-50 of 10000 results (38ms)
2026-02-12 §
14:34 <andrewbogott> moritz is upgrading dnsmasq on eqiad1 cloudnets and cloudvirts [admin]
10:08 <dhinus> delete job "updatetools" in admin tool as it's no longer used (T415383) [admin]
2026-02-09 §
16:03 <andrewbogott> rebooting cloudnets in codfw1dev to make sure we've picked up the new dnsmasq version [admin]
12:50 <dcaro> removed stall cert cloudinfra-acme-chief-01.novalocal from cloudinfra-internal-puppetserver-1 [admin]
12:49 <dcaro> removed the pki* certs from the cloudinfra-cloudvps puppetserver as they are not handled there anymore but the local puppetserver [admin]
09:45 <dcaro> re-enabling nrpe2nodexp-ferm_active.service on cloudcumins after upgrade (getting stall promfile) [admin]
03:00 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services [admin]
02:47 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services [admin]
02:47 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster (exit_code=0) on deployment eqiad1 [admin]
02:44 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster on deployment eqiad1 [admin]
2026-02-05 §
21:12 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services [admin]
21:07 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [admin]
21:03 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster (exit_code=0) on deployment codfw1dev [admin]
21:01 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster on deployment codfw1dev [admin]
20:58 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster (exit_code=99) on deployment codfw1dev [admin]
20:57 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster on deployment codfw1dev [admin]
20:56 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster (exit_code=99) on deployment codfw1dev [admin]
20:55 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster on deployment codfw1dev [admin]
20:49 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster (exit_code=99) on deployment codfw1dev [admin]
20:48 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster on deployment codfw1dev [admin]
20:47 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster (exit_code=99) on deployment codfw1dev [admin]
20:46 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster on deployment codfw1dev [admin]
20:46 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster (exit_code=99) on deployment codfw1dev [admin]
20:46 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster on deployment codfw1dev [admin]
20:45 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster (exit_code=99) on deployment codfw1dev [admin]
20:44 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster on deployment codfw1dev [admin]
20:43 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster (exit_code=99) on deployment codfw1dev [admin]
20:43 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster on deployment codfw1dev [admin]
2026-02-03 §
18:02 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,designate [admin]
18:01 <taavi@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,designate [admin]
2026-01-24 §
17:28 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services [admin]
17:14 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services [admin]
17:09 <andrewbogott> preemptively rebuilding rabbitmq for eqiad1; message flakiness [admin]
2026-01-23 §
13:02 <taavi> switch https://gitlab.wikimedia.org/repos/cloud/wmcs/utils to fast-forward only merging mode [admin]
2026-01-22 §
18:33 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2006-dev.codfw.wmnet' [admin]
18:27 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2006-dev.codfw.wmnet' [admin]
18:08 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet' [admin]
18:01 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2005-dev.codfw.wmnet' [admin]
17:40 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet' [admin]
17:33 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet' [admin]
17:14 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2005-dev.codfw.wmnet' [admin]
17:04 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' [admin]
17:03 <andrew@cloudcumin1001> END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97) on host 'cloudnet2006-dev.codfw.wmnet' [admin]
17:03 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet' [admin]
15:56 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet' [admin]
15:47 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet' [admin]
14:33 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services [admin]
14:30 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [admin]
14:25 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2004-dev.codfw.wmnet' [admin]
14:17 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet' [admin]