1-50 of 10000 results (1ms)
2026-10-19 §
05:48 <marostegui@cumin1004> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1023.eqiad.wmnet with reason: Testing [production]
05:34 <kartik@deploy1003> helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply [production]
05:29 <kartik@deploy1003> helmfile [eqiad] START helmfile.d/services/machinetranslation: apply [production]
05:26 <kartik@deploy1003> helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply [production]
05:22 <kartik@deploy1003> helmfile [codfw] START helmfile.d/services/machinetranslation: apply [production]
04:08 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph} AND NOT P{F:kernelversion = 6.12.113}' [admin]
02:00 <mwpresync@deploy1003> Started scap build-images: Publishing wmf/next image [production]
00:00 <root@deploy1003> helmfile [staging] DONE helmfile.d/services/shellbox: sync [production]
00:00 <root@deploy1003> helmfile [staging] START helmfile.d/services/shellbox: sync [production]
2026-10-18 §
23:02 <wmbot~difronzo@tools-bastion-15> deployed v0.11.4 (by DiFronzo on GitHub) [tools.wikivisage]
23:00 <wmbot~difronzo@tools-bastion-15> deployed face-service-v0.11.4 (by DiFronzo on GitHub) [tools.wikivisage-face]
21:51 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph} AND NOT P{F:kernelversion = 6.12.113}' [admin]
19:36 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph} AND NOT P{F:kernelversion = 6.12.113}' [admin]
19:35 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph} AND NOT P{F:kernelversion = 6.12.113}' [admin]
18:08 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph} AND NOT P{F:kernelversion = 6.12.113}' [admin]
14:08 <andrewbogott> re-deploying paws -- T441568 [paws]
13:54 <andrewbogott> hub-67755dbcd7-ssztz in hopes that its replacement will work better -- T441568 [paws]
13:50 <andrewbogott> killing ingress-nginx-controller in hopes that its replacement will work better [paws]
13:38 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph} AND NOT P{F:kernelversion = 6.12.113}' [admin]
12:00 <root@deploy1003> helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [production]
12:00 <root@deploy1003> helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [production]
08:40 <wmbot~lucaswerkmeister@tools-bastion-15> deployed a32ca0a: Add a tiny bit of information to the README [tools.song-quiz]
05:22 <andrew@cloudcumin1001> END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph} AND NOT P{F:kernelversion = 6.12.113}' [admin]
03:43 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph} AND NOT P{F:kernelversion = 6.12.113}' [admin]
03:42 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph} AND NOT P{F:kernelversion = 6.12.113}' [admin]
02:40 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph} AND NOT P{F:kernelversion = 6.12.113}' [admin]
02:40 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P{P:openstack::eqiad1::nova::compute::service} AND NOT P{F:kernelversion = 6.12.113}' [admin]
02:37 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{P:openstack::eqiad1::nova::compute::service} AND NOT P{F:kernelversion = 6.12.113}' [admin]
02:37 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1089.eqiad.wmnet' [admin]
02:36 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1089.eqiad.wmnet' [admin]
02:36 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1088.eqiad.wmnet' [admin]
02:36 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1088.eqiad.wmnet' [admin]
02:36 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1087.eqiad.wmnet' [admin]
02:36 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1087.eqiad.wmnet' [admin]
02:36 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1086.eqiad.wmnet' [admin]
02:36 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1086.eqiad.wmnet' [admin]
02:36 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1085.eqiad.wmnet' [admin]
02:36 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1085.eqiad.wmnet' [admin]
02:36 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1084.eqiad.wmnet' [admin]
02:36 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1084.eqiad.wmnet' [admin]
02:36 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1083.eqiad.wmnet' [admin]
02:36 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1083.eqiad.wmnet' [admin]
02:36 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1082.eqiad.wmnet' [admin]
02:36 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1082.eqiad.wmnet' [admin]
02:36 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1081.eqiad.wmnet' [admin]
02:36 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1081.eqiad.wmnet' [admin]
02:36 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1080.eqiad.wmnet' [admin]
02:31 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1080.eqiad.wmnet' [admin]
02:26 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1079.eqiad.wmnet' [admin]
02:22 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1079.eqiad.wmnet' [admin]