1-50 of 10000 results (0ms)
2026-09-29 §
05:54 <ryankemper@cumin2003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1035.eqiad.wmnet with OS bookworm [production]
05:50 <ryankemper@cumin2003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1034.eqiad.wmnet with OS bookworm [production]
05:38 <ryankemper@cumin2003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1035.eqiad.wmnet with reason: host reimage [production]
05:34 <ryankemper@cumin2003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1034.eqiad.wmnet with reason: host reimage [production]
05:32 <ryankemper@cumin2003> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1035.eqiad.wmnet with reason: host reimage [production]
05:31 <ryankemper@cumin2003> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1034.eqiad.wmnet with reason: host reimage [production]
05:21 <ryankemper@cumin2003> END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1035 [production]
05:21 <ryankemper@cumin2003> START - Cookbook sre.hosts.move-vlan for host wdqs1035 [production]
05:21 <ryankemper@cumin2003> START - Cookbook sre.hosts.reimage for host wdqs1035.eqiad.wmnet with OS bookworm [production]
05:19 <ryankemper@cumin2003> END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1034 [production]
05:19 <ryankemper@cumin2003> START - Cookbook sre.hosts.move-vlan for host wdqs1034 [production]
05:19 <ryankemper@cumin2003> START - Cookbook sre.hosts.reimage for host wdqs1034.eqiad.wmnet with OS bookworm [production]
05:03 <ryankemper@cumin2003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1033.eqiad.wmnet with OS bookworm [production]
04:51 <fabfur> manually restarting dump_cloud_ip_range on puppetserver1001 after failure [production]
04:47 <ryankemper@cumin2003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1033.eqiad.wmnet with reason: host reimage [production]
04:43 <ryankemper@cumin2003> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1033.eqiad.wmnet with reason: host reimage [production]
04:32 <ryankemper@cumin2003> END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1033 [production]
04:32 <ryankemper@cumin2003> START - Cookbook sre.hosts.move-vlan for host wdqs1033 [production]
04:31 <ryankemper@cumin2003> START - Cookbook sre.hosts.reimage for host wdqs1033.eqiad.wmnet with OS bookworm [production]
04:01 <mwpresync@deploy1003> Pruned MediaWiki: 1.47.0-wmf.19 (duration: 01m 00s) [production]
03:39 <mwpresync@deploy1003> Finished scap sync-world: testwikis to 1.47.0-wmf.22 refs T438218 (duration: 36m 03s) [production]
03:03 <mwpresync@deploy1003> Started scap sync-world: testwikis to 1.47.0-wmf.22 refs T438218 [production]
02:47 <larssandergreen> civicrm upgraded from 11d70b96 to a7d8ba32 [fundraising]
02:08 <mwpresync@deploy1003> Finished scap build-images: Publishing wmf/next image (duration: 07m 28s) [production]
02:00 <mwpresync@deploy1003> Started scap build-images: Publishing wmf/next image [production]
01:56 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [admin]
01:49 <brett@cumin1004> END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_esams - 7.1.1-2~bpo13+wmf3 () [production]
01:46 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add [admin]
01:39 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [admin]
01:39 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add [admin]
01:36 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [admin]
01:36 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add [admin]
01:35 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [admin]
01:35 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add [admin]
00:48 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [admin]
00:22 <dzahn@cumin1004> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage [production]
00:17 <dzahn@cumin1004> START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage [production]
00:01 <dzahn@cumin1004> START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie [production]
2026-09-28 §
23:44 <dzahn@cumin1004> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie [production]
23:35 <zabe> run script to fix fr_metadata drifts # T420341 [production]
22:22 <lerickson@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [production]
22:21 <lerickson@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [production]
21:47 <sbassett> Deployed security mitigation for T435455 [production]
21:44 <brett@cumin1004> START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_esams - 7.1.1-2~bpo13+wmf3 () [production]
21:43 <brett@cumin1004> END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_esams - 7.1.1-2~bpo13+wmf3 () [production]
21:42 <eevans@deploy1003> helmfile [eqiad] DONE helmfile.d/services/sessionstore: sync [production]
21:42 <eevans@deploy1003> helmfile [eqiad] START helmfile.d/services/sessionstore: sync [production]
21:41 <eevans@deploy1003> helmfile [staging] DONE helmfile.d/services/sessionstore: sync [production]
21:41 <eevans@deploy1003> helmfile [staging] START helmfile.d/services/sessionstore: sync [production]
21:24 <swfrench-wmf> deleting suspect coredns pods showing upstream resolution health check failures in eqiad - T439201 [production]