101-150 of 10000 results (102ms)
2026-05-05 ยง
19:17 <dancy@deploy1003> Installation of scap version "4.262.0" completed for 2 hosts [production]
19:15 <dancy@deploy1003> Installing scap version "4.262.0" for 2 host(s) [production]
19:15 <cmooney@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 6 hosts with reason: rebooting firewall in desperation [production]
19:14 <brennen@deploy1003> rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.1 refs T423910 [production]
19:05 <cmooney@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "set correct vlan group in netbox for new ulsfo vlans - cmooney@cumin1003 - T408892" [production]
19:05 <cmooney@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "set correct vlan group in netbox for new ulsfo vlans - cmooney@cumin1003 - T408892" [production]
19:04 <pt1979@cumin1003> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4038.ulsfo.wmnet with OS trixie [production]
19:02 <ladsgroup@deploy1003> Finished scap sync-world: Backport for [[gerrit:1283063|Close Swedish Wikinews (T421796)]] (duration: 10m 59s) [production]
18:56 <ladsgroup@deploy1003> ladsgroup: Continuing with deployment [production]
18:55 <ladsgroup@deploy1003> ladsgroup: Backport for [[gerrit:1283063|Close Swedish Wikinews (T421796)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
18:52 <ladsgroup@deploy1003> Started scap sync-world: Backport for [[gerrit:1283063|Close Swedish Wikinews (T421796)]] [production]
18:49 <pt1979@cumin1003> START - Cookbook sre.hosts.reimage for host cp4038.ulsfo.wmnet with OS trixie [production]
18:48 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:48 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update cp4038 ip address - pt1979@cumin2002" [production]
18:48 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update cp4038 ip address - pt1979@cumin2002" [production]
18:47 <brennen@deploy1003> Finished scap sync-world: testwikis to 1.47.0-wmf.1 refs T423910 (duration: 36m 04s) [production]
18:44 <pt1979@cumin1003> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4038.ulsfo.wmnet with OS trixie [production]
18:44 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
18:44 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update cp4038 ip address - pt1979@cumin2002" [production]
18:44 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:44 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update cp4038 ip address - pt1979@cumin2002" [production]
18:40 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
18:30 <pt1979@cumin1003> START - Cookbook sre.hosts.reimage for host cp4038.ulsfo.wmnet with OS trixie [production]
18:25 <pt1979@cumin1003> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4038.ulsfo.wmnet with OS trixie [production]
18:14 <cmooney@cumin1003> END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device pfw1a-codfw [production]
18:13 <cmooney@cumin1003> START - Cookbook sre.network.tls for network device pfw1a-codfw [production]
18:13 <pt1979@cumin1003> START - Cookbook sre.hosts.reimage for host cp4038.ulsfo.wmnet with OS trixie [production]
18:11 <brennen@deploy1003> Started scap sync-world: testwikis to 1.47.0-wmf.1 refs T423910 [production]
18:10 <cmooney@cumin1003> END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device pfw1a-eqiad [production]
18:10 <cmooney@cumin1003> START - Cookbook sre.network.tls for network device pfw1a-eqiad [production]
18:06 <brennen> 1.47.0-wmf.1 train status (T423910): no current blockers, rolling to group0 [production]
18:04 <herron@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging1003.eqiad.wmnet with OS trixie [production]
17:44 <herron@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging1003.eqiad.wmnet with reason: host reimage [production]
17:38 <herron@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging1003.eqiad.wmnet with reason: host reimage [production]
17:33 <herron@deploy1003> helmfile [codfw] DONE helmfile.d/admin 'sync'. [production]
17:32 <herron@deploy1003> helmfile [codfw] START helmfile.d/admin 'sync'. [production]
17:23 <root@deploy1003> helmfile [eqiad] DONE helmfile.d/admin 'sync'. [production]
17:23 <root@deploy1003> helmfile [eqiad] START helmfile.d/admin 'sync'. [production]
17:21 <herron@cumin1003> END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host kafka-logging1003 [production]
17:21 <herron@cumin1003> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-logging1003 [production]
17:21 <herron@cumin1003> START - Cookbook sre.network.configure-switch-interfaces for host kafka-logging1003 [production]
17:20 <herron@cumin1003> END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) kafka-logging1003.eqiad.wmnet 66.48.64.10.in-addr.arpa 6.6.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [production]
17:19 <herron@cumin1003> START - Cookbook sre.dns.wipe-cache kafka-logging1003.eqiad.wmnet 66.48.64.10.in-addr.arpa 6.6.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [production]
17:16 <herron@cumin1003> END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) kafka-logging1003.eqiad.wmnet 66.48.64.10.in-addr.arpa 6.6.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [production]
17:15 <herron@cumin1003> START - Cookbook sre.dns.wipe-cache kafka-logging1003.eqiad.wmnet 66.48.64.10.in-addr.arpa 6.6.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [production]
17:15 <herron@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:15 <herron@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging1003 - herron@cumin1003" [production]
17:15 <herron@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host kafka-logging1003 - herron@cumin1003" [production]
17:12 <herron@cumin1003> START - Cookbook sre.dns.netbox [production]
17:09 <herron@cumin1003> START - Cookbook sre.hosts.move-vlan for host kafka-logging1003 [production]