5851-5900 of 10000 results (141ms)
2024-02-28 §
00:07 <rzl@deploy2002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
00:06 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 15:00:00 on wdqs1011.eqiad.wmnet with reason: T355617 [production]
00:06 <bking@cumin2002> START - Cookbook sre.hosts.downtime for 15:00:00 on wdqs1011.eqiad.wmnet with reason: T355617 [production]
00:02 <dzahn@cumin1002> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host contint1003.eqiad.wmnet [production]
00:02 <dzahn@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host contint1003.eqiad.wmnet with OS bullseye [production]
00:02 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2199.codfw.wmnet with OS bookworm [production]
00:01 <jhancock@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" [production]
00:00 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2202.codfw.wmnet with OS bookworm [production]
00:00 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" [production]
2024-02-27 §
23:57 <mutante> T358237 - manually went through "fix forward"-steps from T349619 (install puppet-agent package, delete old key material, create new CSR, sign on puppetserver, node clean on puppetmaster) to fix puppet failures while makevm cookbook still running (which couldn't find succesful puppet run) [production]
23:54 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" [production]
23:54 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2201.codfw.wmnet with OS bookworm [production]
23:54 <jhancock@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" [production]
23:52 <mutante> T358237 - creating VM with cookbook fails because puppet runs have certificate issue, applied role is already migrated to puppet 7 though [production]
23:50 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" [production]
23:49 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2200.codfw.wmnet with OS bookworm [production]
23:49 <jhancock@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" [production]
23:45 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" [production]
23:40 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2199.codfw.wmnet with reason: host reimage [production]
23:38 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2202.codfw.wmnet with reason: host reimage [production]
23:36 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2201.codfw.wmnet with reason: host reimage [production]
23:33 <jhancock@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2199.codfw.wmnet with reason: host reimage [production]
23:33 <jhancock@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2202.codfw.wmnet with reason: host reimage [production]
23:33 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2200.codfw.wmnet with reason: host reimage [production]
23:33 <jhancock@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2201.codfw.wmnet with reason: host reimage [production]
23:30 <jhancock@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2200.codfw.wmnet with reason: host reimage [production]
23:10 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host db2202.codfw.wmnet with OS bookworm [production]
23:10 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host db2201.codfw.wmnet with OS bookworm [production]
23:10 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host db2200.codfw.wmnet with OS bookworm [production]
23:10 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host db2199.codfw.wmnet with OS bookworm [production]
23:10 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host db2198.codfw.wmnet with OS bookworm [production]
23:09 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host db2197.codfw.wmnet with OS bookworm [production]
22:47 <mutante> DNS - added new project language "bew" - Betawi, also known as Betawi Malay, Jakartan Malay, or Batavian Malay is the spoken language of the Betawi people in Jakarta, Indonesia with an estimated 5 million native speakers. T357866 [production]
22:44 <dzahn@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on contint1003.eqiad.wmnet with reason: host reimage [production]
22:41 <dzahn@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on contint1003.eqiad.wmnet with reason: host reimage [production]
22:32 <dzahn@cumin1002> START - Cookbook sre.hosts.reimage for host contint1003.eqiad.wmnet with OS bullseye [production]
22:31 <dzahn@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM contint1003.eqiad.wmnet - dzahn@cumin1002" [production]
22:30 <dzahn@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM contint1003.eqiad.wmnet - dzahn@cumin1002" [production]
22:30 <dzahn@cumin1002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) contint1003.eqiad.wmnet on all recursors [production]
22:30 <dzahn@cumin1002> START - Cookbook sre.dns.wipe-cache contint1003.eqiad.wmnet on all recursors [production]
22:30 <dzahn@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:30 <dzahn@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM contint1003.eqiad.wmnet - dzahn@cumin1002" [production]
22:29 <dzahn@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM contint1003.eqiad.wmnet - dzahn@cumin1002" [production]
22:24 <dzahn@cumin1002> START - Cookbook sre.dns.netbox [production]
22:24 <dzahn@cumin1002> START - Cookbook sre.ganeti.makevm for new host contint1003.eqiad.wmnet [production]
20:51 <cdanis@deploy2002> helmfile [aux-k8s-eqiad] DONE helmfile.d/aus-k8s-eqiad-services/jaeger: sync [production]
20:51 <cdanis@deploy2002> helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: sync [production]
20:50 <cdanis@deploy2002> helmfile [aux-k8s-eqiad] DONE helmfile.d/aus-k8s-eqiad-services/jaeger: apply [production]
20:50 <cdanis@deploy2002> helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: apply [production]
20:48 <cdanis@deploy2002> helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: sync [production]