201-250 of 10000 results (4ms)
2026-08-26 ยง
16:48 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-2 from 1.32.13 to 1.33.13 (T433132) [tools]
16:47 <cmooney@cumin1003> START - Cookbook sre.dns.netbox [production]
16:47 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- [tools]
16:46 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- [tools]
16:46 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- [tools]
16:46 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 (T433132) [tools]
16:44 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-1 from 1.32.13 to 1.33.13 (T433132) [tools]
16:43 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-1 from 1.32.13 to 1.33.13 (T433132) [tools]
16:33 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 (T433132) [tools]
16:30 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.32.13 to 1.33.13 (T433132) [tools]
16:25 <cmooney@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 12 hosts with reason: site migration to new Nokia switches [production]
16:25 <cmooney@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 13 hosts with reason: site migration to new Nokia switches [production]
16:24 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.32.13 to 1.33.13 (T433132) [tools]
16:24 <cmooney@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 12 hosts with reason: site migration to new Nokia switches [production]
16:24 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.32.13 to 1.33.13 (T433132) [tools]
16:23 <cmooney@cumin1003> END (PASS) - Cookbook sre.metamonitoring.downtime (exit_code=0) Downtime for 4:00:00 of prometheus/deadmanswitchnotified, prometheus/deadmanswitchonamdb, prometheus/extmon on 2 host(s) with reason: eqsin site rebuild [production]
16:23 <cmooney@cumin1003> START - Cookbook sre.metamonitoring.downtime Downtime for 4:00:00 of prometheus/deadmanswitchnotified, prometheus/deadmanswitchonamdb, prometheus/extmon on 2 host(s) with reason: eqsin site rebuild [production]
16:20 <cdobbins@cumin1003> conftool action : set/pooled=yes; selector: name=dns1004.* [reason: trixie upgrade] [production]
16:19 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.32.13 to 1.33.13 (T433132) [tools]
16:18 <cdobbins@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns1004.wikimedia.org with OS trixie [production]
15:48 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=0) for tools-k8s-control-7 (T433132) [tools]
15:44 <cmooney@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:44 <cmooney@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix dns entries for cr2-eqiad transport interface IPs - cmooney@cumin1003" [production]
15:44 <cdobbins@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns1004.wikimedia.org with reason: host reimage [production]
15:42 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.restart_static_pods for tools-k8s-control-7 (T433132) [tools]
15:41 <cmooney@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix dns entries for cr2-eqiad transport interface IPs - cmooney@cumin1003" [production]
15:38 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-7 from 1.32.13 to 1.33.13 (T433132) [tools]
15:37 <cdobbins@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on dns1004.wikimedia.org with reason: host reimage [production]
15:35 <cmooney@cumin1003> START - Cookbook sre.dns.netbox [production]
15:33 <lerickson@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [production]
15:33 <cmooney@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:33 <cmooney@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix dns entries for cr2-eqiad transport interface IPs - cmooney@cumin1003" [production]
15:33 <cmooney@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix dns entries for cr2-eqiad transport interface IPs - cmooney@cumin1003" [production]
15:31 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.32.13 to 1.33.13 (T433132) [tools]
15:31 <lerickson@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [production]
15:30 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-7 from 1.32.13 to 1.33.13 (T433132) [tools]
15:28 <cmooney@cumin1003> START - Cookbook sre.dns.netbox [production]
15:28 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.32.13 to 1.33.13 (T433132) [tools]
15:27 <cmooney@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:25 <cmooney@cumin1003> START - Cookbook sre.dns.netbox [production]
15:24 <mpostoronca> Deployed patch for T432848 [production]
15:21 <cdobbins@cumin1003> START - Cookbook sre.hosts.reimage for host dns1004.wikimedia.org with OS trixie [production]
15:20 <cdobbins@cumin1003> conftool action : set/pooled=no; selector: name=dns1004.* [reason: trixie upgrade] [production]
15:20 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-7 from 1.32.13 to 1.33.13 (T433132) [tools]
15:19 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.32.13 to 1.33.13 (T433132) [tools]
15:17 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade to 1.33.13 (T433132) [tools]
15:10 <fceratto@cumin1003> END (PASS) - Cookbook sre.mysql.decommission (exit_code=0) [production]
15:10 <fceratto@cumin1003> Removing db-test2001 from zarcillo T435912 [production]
15:02 <bking@cumin2003> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host krb1004.eqiad.wmnet [production]
15:02 <bking@cumin2003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host krb1004.eqiad.wmnet with OS bookworm [production]