1-50 of 10000 results (25ms)
2026-08-28 §
11:23 <jmm@cumin2003> START - Cookbook sre.dns.netbox [production]
11:23 <jmm@cumin2003> START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet [production]
11:12 <aokoth@cumin1003> END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - T436069 [production]
10:43 <aokoth@cumin1003> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - T436069 [production]
08:59 <jmm@cumin2003> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet [production]
08:59 <jmm@cumin2003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:59 <jmm@cumin2003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" [production]
08:58 <jmm@cumin2003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" [production]
08:28 <jmm@cumin2003> START - Cookbook sre.dns.netbox [production]
07:49 <jmm@cumin2003> START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet [production]
06:30 <jmm@cumin2003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org [production]
06:26 <jmm@cumin2003> START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org [production]
04:34 <ryankemper> T435862 Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done [production]
04:31 <ryankemper@cumin2003> END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. [production]
03:59 <ryankemper@cumin2003> START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. [production]
03:58 <ryankemper> T435862 Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined [production]
03:42 <ryankemper> T435862 Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded [production]
03:40 <ryankemper@cumin2003> END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. [production]
03:38 <ryankemper@cumin2003> START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. [production]
03:10 <tstarling@deploy1003> Finished scap sync-world: Backport for [[gerrit:1328752|Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728|Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) [production]
03:05 <tstarling@deploy1003> tstarling: Continuing with deployment [production]
03:02 <tstarling@deploy1003> tstarling: Backport for [[gerrit:1328752|Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728|Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
02:58 <tstarling@deploy1003> Started scap sync-world: Backport for [[gerrit:1328752|Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728|Produnto: Add IPv6 range for GitLab (T421436)]] [production]
02:25 <ryankemper> `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) [production]
02:08 <mwpresync@deploy1003> Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) [production]
02:00 <mwpresync@deploy1003> Started scap build-images: Publishing wmf/next image [production]
00:55 <ladsgroup@deploy1003> helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [production]
00:54 <ladsgroup@deploy1003> helmfile [eqiad] START helmfile.d/services/thumbor: apply [production]
00:53 <ladsgroup@deploy1003> helmfile [codfw] DONE helmfile.d/services/thumbor: apply [production]
00:53 <ladsgroup@deploy1003> helmfile [codfw] START helmfile.d/services/thumbor: apply [production]
00:53 <ladsgroup@deploy1003> helmfile [staging] DONE helmfile.d/services/thumbor: apply [production]
00:52 <ladsgroup@deploy1003> helmfile [staging] START helmfile.d/services/thumbor: apply [production]
00:40 <ladsgroup@deploy1003> helmfile [staging] DONE helmfile.d/services/thumbor: apply [production]
00:39 <ladsgroup@deploy1003> helmfile [staging] START helmfile.d/services/thumbor: apply [production]
00:27 <ladsgroup@deploy1003> helmfile [codfw] DONE helmfile.d/services/thumbor: apply [production]
00:27 <ladsgroup@deploy1003> helmfile [codfw] START helmfile.d/services/thumbor: apply [production]
00:24 <ladsgroup@deploy1003> helmfile [staging] DONE helmfile.d/services/thumbor: apply [production]
00:23 <ladsgroup@deploy1003> helmfile [staging] START helmfile.d/services/thumbor: apply [production]
2026-08-27 §
23:14 <jasmine@cumin1003> END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet [production]
23:14 <jasmine@cumin1003> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet [production]
23:14 <jasmine@cumin1003> START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet [production]
23:07 <jasmine_> homer lsw1-c6-eqiad* commit 'T421711' [production]
23:05 <jasmine@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie [production]
22:44 <jasmine@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage [production]
22:40 <jasmine@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage [production]
22:20 <jasmine@cumin1003> END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 [production]
22:20 <jasmine@cumin1003> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 [production]
22:19 <jasmine@cumin1003> START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 [production]
22:19 <jasmine@cumin1003> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [production]
22:19 <jasmine@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]