1-50 of 10000 results (4ms)
2026-09-30 §
09:08 <btullis@dns1004> END - running authdns-update [production]
09:06 <btullis@dns1004> START - running authdns-update [production]
09:04 <elukey> elukey@cumin1004:~$ sudo cumin 'A:lvs-low-traffic-codfw' 'systemctl restart pybal.service' [production]
09:02 <dcaro@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [tools]
09:01 <elukey> elukey@cumin1004:~$ sudo cumin 'A:lvs-secondary-codfw' 'systemctl restart pybal.service' [production]
09:00 <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [tools]
08:55 <dcaro@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [tools]
08:51 <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [tools]
08:44 <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [toolsbeta]
08:30 <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [toolsbeta]
08:29 <dcaro@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [toolsbeta]
08:29 <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [toolsbeta]
08:27 <dcaro@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [toolsbeta]
08:27 <jmm@cumin2003> END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-eqiad [production]
08:27 <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [toolsbeta]
08:25 <jmm@cumin2003> START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-eqiad [production]
08:24 <elukey@cumin1004> conftool action : set/pooled=yes:weight=1; selector: cluster=pki,service=cfssl-multirootca [production]
08:24 <jmm@cumin2003> END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-codfw [production]
08:22 <jmm@cumin2003> START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-codfw [production]
08:07 <dcaro@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [toolsbeta]
07:58 <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [toolsbeta]
07:49 <filippo@cumin1004> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm [production]
07:46 <elukey@dns1004> END - running authdns-update [production]
07:44 <kevinbazira@deploy1003> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . [production]
07:44 <kevinbazira@deploy1003> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
07:44 <elukey@dns1004> START - running authdns-update [production]
07:35 <moritzm> installing pcre2/openssl security updates [production]
07:18 <filippo@cumin1004> START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm [production]
07:18 <filippo@cumin1004> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm [production]
07:15 <dcausse@deploy1003> Finished scap sync-world: Backport for [[gerrit:1346012|Search: optimize morelike queries]] (duration: 11m 27s) [production]
07:13 <filippo@cumin1004> START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm [production]
07:12 <filippo@cumin1004> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm [production]
07:09 <dcausse@deploy1003> dcausse: Continuing with deployment [production]
07:07 <dcausse@deploy1003> dcausse: Backport for [[gerrit:1346012|Search: optimize morelike queries]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
07:03 <dcausse@deploy1003> Started scap sync-world: Backport for [[gerrit:1346012|Search: optimize morelike queries]] [production]
06:51 <filippo@cumin1004> START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm [production]
06:48 <filippo@cumin1004> END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cloudcephosd1056.eqiad.wmnet [production]
06:33 <filippo@cumin1004> START - Cookbook sre.hosts.dhcp for host cloudcephosd1056.eqiad.wmnet [production]
06:32 <filippo@cumin1004> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm [production]
02:20 <eileen> civicrm upgraded from 6144ab57 to 7ff3e786 [fundraising]
02:08 <mwpresync@deploy1003> Finished scap build-images: Publishing wmf/next image (duration: 07m 40s) [production]
02:00 <mwpresync@deploy1003> Started scap build-images: Publishing wmf/next image [production]
01:28 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [admin]
01:25 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [admin]
01:16 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add [admin]
2026-09-29 §
22:42 <bking@cumin2003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1044.eqiad.wmnet with OS bookworm [production]
22:39 <bking@cumin2003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1043.eqiad.wmnet with OS bookworm [production]
22:26 <bking@cumin2003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1044.eqiad.wmnet with reason: host reimage [production]
22:23 <bking@cumin2003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1043.eqiad.wmnet with reason: host reimage [production]
22:21 <bking@cumin2003> START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1044.eqiad.wmnet with reason: host reimage [production]