3001-3050 of 10000 results (100ms)
2023-03-14 ยง
19:44 <cmooney@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on cloudsw1-b1-codfw,cloudsw1-b1-codfw IPv6,cloudsw1-b1-codfw.mgmt with reason: cloudsw1-b1-codfw OS upgrade [production]
19:44 <cmooney@cumin1001> START - Cookbook sre.hosts.downtime for 0:30:00 on cloudsw1-b1-codfw,cloudsw1-b1-codfw IPv6,cloudsw1-b1-codfw.mgmt with reason: cloudsw1-b1-codfw OS upgrade [production]
19:32 <jhathaway@cumin2002> START - Cookbook sre.hosts.reimage for host sretest1002.eqiad.wmnet with OS bookworm [production]
19:30 <brennen> 1.40.0-wmf.27 train (T330205): uneventful at group0. i'm afk for about an hour. [production]
19:13 <ejegg> civicrm upgraded from dbe3b716 to 68fa85cf [production]
18:51 <herron@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging2002.codfw.wmnet with OS bullseye [production]
18:32 <herron@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging2002.codfw.wmnet with reason: host reimage [production]
18:28 <fab@deploy2002> Finished deploy [airflow-dags/research@5edcd7b]: (no justification provided) (duration: 00m 11s) [production]
18:27 <fab@deploy2002> Started deploy [airflow-dags/research@5edcd7b]: (no justification provided) [production]
18:27 <herron@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging2002.codfw.wmnet with reason: host reimage [production]
18:25 <hnowlan@deploy2002> helmfile [staging] DONE helmfile.d/services/device-analytics: apply [production]
18:25 <hnowlan@deploy2002> helmfile [staging] START helmfile.d/services/device-analytics: apply [production]
18:25 <hnowlan@deploy2002> helmfile [staging] DONE helmfile.d/services/device-analytics: apply [production]
18:22 <fab@deploy2002> Finished deploy [airflow-dags/research@5edcd7b]: (no justification provided) (duration: 00m 30s) [production]
18:22 <fab@deploy2002> Started deploy [airflow-dags/research@5edcd7b]: (no justification provided) [production]
18:15 <hnowlan@deploy2002> helmfile [staging] START helmfile.d/services/device-analytics: apply [production]
18:13 <brennen@deploy2002> rebuilt and synchronized wikiversions files: group0 wikis to 1.40.0-wmf.27 refs T330205 [production]
18:13 <herron@cumin1001> START - Cookbook sre.hosts.reimage for host kafka-logging2002.codfw.wmnet with OS bullseye [production]
18:06 <hnowlan@deploy2002> helmfile [staging] DONE helmfile.d/services/device-analytics: apply [production]
18:06 <hnowlan@deploy2002> helmfile [staging] START helmfile.d/services/device-analytics: apply [production]
18:03 <brennen> 1.40.0-wmf.27 train (T330205): no current blockers, rolling to group0. [production]
17:59 <hnowlan@deploy2002> helmfile [codfw] DONE helmfile.d/admin 'apply'. [production]
17:59 <hnowlan@deploy2002> helmfile [codfw] START helmfile.d/admin 'apply'. [production]
17:58 <hnowlan@deploy2002> helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [production]
17:56 <hnowlan@deploy2002> helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [production]
17:55 <hnowlan@deploy2002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
17:55 <hnowlan@deploy2002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
17:53 <hnowlan@deploy2002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
17:52 <hnowlan@deploy2002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
17:52 <hnowlan@deploy2002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
17:52 <hnowlan@deploy2002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
17:11 <aborrero@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudlb2003-dev.codfw.wmnet with OS bullseye [production]
17:08 <aborrero@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudlb2002-dev.codfw.wmnet with OS bullseye [production]
16:49 <mvernon@cumin2002> START - Cookbook sre.hosts.reboot-single for host ms-be2067.codfw.wmnet [production]
16:47 <sukhe> rolling restart of pdns-rec in A:wikidough to pick up config changes [production]
16:47 <sukhe> rolling restart of pdns-rec to pick up config changes [production]
16:44 <gmodena@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
16:44 <gmodena@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply [production]
16:16 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pki2001.codfw.wmnet [production]
16:16 <jbond@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:16 <jbond@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pki2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jbond@cumin1001" [production]
16:13 <jbond@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pki2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jbond@cumin1001" [production]
16:11 <jbond@cumin1001> START - Cookbook sre.dns.netbox [production]
16:04 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 12:00:00 on cephosd[1001-1005].eqiad.wmnet with reason: Bootstrapping ceph [production]
16:04 <btullis@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 12:00:00 on cephosd[1001-1005].eqiad.wmnet with reason: Bootstrapping ceph [production]
16:00 <jbond@cumin1001> START - Cookbook sre.hosts.decommission for hosts pki2001.codfw.wmnet [production]
15:59 <herron@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging2003.codfw.wmnet with OS bullseye [production]
15:36 <herron@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging2003.codfw.wmnet with reason: host reimage [production]
15:35 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on vrts2001.codfw.wmnet with reason: installation failed due to read-only database [production]
15:35 <aokoth@cumin1001> START - Cookbook sre.hosts.downtime for 14 days, 0:00:00 on vrts2001.codfw.wmnet with reason: installation failed due to read-only database [production]