1-50 of 10000 results (18ms)
2026-08-13 §
07:31 <marostegui@cumin1003> END (ERROR) - Cookbook sre.mysql.sanitize-wiki (exit_code=97) Managing sanitization for wikis testwiki in section s3 [production]
07:29 <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1049' (T431682) [admin]
07:24 <marostegui@cumin1003> START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis testwiki in section s3 [production]
07:19 <jayme@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on kubestagemaster2005.codfw.wmnet with reason: downtime because of hardware failure and no DRBD [production]
05:42 <arnaudb@dns1006> END - running authdns-update [production]
05:40 <arnaudb@dns1006> START - running authdns-update [production]
05:27 <vriley@cumin1003> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1006.eqiad.wmnet with OS trixie [production]
04:06 <vriley@cumin1003> START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie [production]
03:05 <TimStarling> created Produnto tables on beta T421436 [releng]
02:29 <ryankemper@cumin2003> END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1151 [production]
02:29 <ryankemper@cumin2003> START - Cookbook sre.hosts.move-vlan for host an-worker1151 [production]
02:23 <ryankemper@cumin2003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1219.eqiad.wmnet with OS bookworm [production]
02:18 <ryankemper> T434494 `ryankemper@deploy1003:~$ echo 'https://stats.wikimedia.org/' | mwscript-k8s --attach -- purgeList.php` (default page got cached during yesterday's `an-web1001` reimage) [production]
02:10 <ejegg> payments-wiki upgraded from fff3c1de to 44194999 [fundraising]
02:07 <mwpresync@deploy1003> Finished scap build-images: Publishing wmf/next image (duration: 06m 46s) [production]
02:04 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-1 from 1.32.13 to 1.33.13 (TT433131) [toolsbeta]
02:03 <ryankemper@cumin2003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1219.eqiad.wmnet with reason: host reimage [production]
02:03 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-1 from 1.32.13 to 1.33.13 (TT433131) [toolsbeta]
02:02 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-2 from 1.32.13 to 1.33.13 (TT433131) [toolsbeta]
02:02 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-2 from 1.32.13 to 1.33.13 (TT433131) [toolsbeta]
02:01 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-3 from 1.32.13 to 1.33.13 (TT433131) [toolsbeta]
02:01 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-3 from 1.32.13 to 1.33.13 (TT433131) [toolsbeta]
02:00 <mwpresync@deploy1003> Started scap build-images: Publishing wmf/next image [production]
02:00 <ryankemper@cumin2003> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1219.eqiad.wmnet with reason: host reimage [production]
01:59 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 (T433131) [toolsbeta]
01:58 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 (T433131) [toolsbeta]
01:57 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 (T433131) [toolsbeta]
01:56 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 (T433131) [toolsbeta]
01:51 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-13 [toolsbeta]
01:50 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-13 [toolsbeta]
01:49 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-13 [toolsbeta]
01:49 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-13 [toolsbeta]
01:48 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) [toolsbeta]
01:48 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) [toolsbeta]
01:47 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) [toolsbeta]
01:46 <ryankemper@cumin2003> START - Cookbook sre.hosts.reimage for host an-worker1219.eqiad.wmnet with OS bookworm [production]
01:17 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) [toolsbeta]
01:03 <ryankemper@cumin2003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1142.eqiad.wmnet with OS bookworm [production]
00:52 <eileen> civicrm upgraded from 0d8a3fcd to d5420716 [fundraising]
00:38 <ryankemper@cumin2003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1142.eqiad.wmnet with reason: host reimage [production]
00:34 <ryankemper@cumin2003> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1142.eqiad.wmnet with reason: host reimage [production]
00:19 <ryankemper@cumin2003> START - Cookbook sre.hosts.reimage for host an-worker1142.eqiad.wmnet with OS bookworm [production]
00:16 <ryankemper@cumin2003> END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1178 [production]
00:16 <ryankemper@cumin2003> START - Cookbook sre.hosts.move-vlan for host an-worker1178 [production]
2026-08-12 §
23:18 <ladsgroup@deploy1003> Finished scap sync-world: Backport for [[gerrit:1324409|Remove $wmg = $wg hacks in Collection (T119117)]] (duration: 06m 43s) [production]
23:14 <ladsgroup@deploy1003> ladsgroup: Continuing with deployment [production]
23:13 <ladsgroup@deploy1003> ladsgroup: Backport for [[gerrit:1324409|Remove $wmg = $wg hacks in Collection (T119117)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
23:11 <ladsgroup@deploy1003> Started scap sync-world: Backport for [[gerrit:1324409|Remove $wmg = $wg hacks in Collection (T119117)]] [production]
22:59 <vriley@cumin1003> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1006.eqiad.wmnet with OS trixie [production]
22:49 <Dreamy_Jazz> Running `mwscript-k8s WikimediaAntiAbuse:BackfillAbuseReview.php --wiki=enwiki --start-timestamp="20260801000000" --end-timestamp="20260802000000" --sleep=2 --batch-size=10` [production]