|
2023-03-14
ยง
|
| 20:04 |
<jhathaway@cumin2002> |
START - Cookbook sre.hosts.reimage for host sretest1002.eqiad.wmnet with OS bookworm |
[production] |
| 20:03 |
<jhathaway@cumin2002> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest1002.eqiad.wmnet with OS bookworm |
[production] |
| 19:47 |
<topranks> |
Reboot cloudsw1-b1-codfw to upgrade JunOS version T327919 |
[production] |
| 19:44 |
<cmooney@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on cloudsw1-b1-codfw,cloudsw1-b1-codfw IPv6,cloudsw1-b1-codfw.mgmt with reason: cloudsw1-b1-codfw OS upgrade |
[production] |
| 19:44 |
<cmooney@cumin1001> |
START - Cookbook sre.hosts.downtime for 0:30:00 on cloudsw1-b1-codfw,cloudsw1-b1-codfw IPv6,cloudsw1-b1-codfw.mgmt with reason: cloudsw1-b1-codfw OS upgrade |
[production] |
| 19:32 |
<jhathaway@cumin2002> |
START - Cookbook sre.hosts.reimage for host sretest1002.eqiad.wmnet with OS bookworm |
[production] |
| 19:30 |
<brennen> |
1.40.0-wmf.27 train (T330205): uneventful at group0. i'm afk for about an hour. |
[production] |
| 19:13 |
<ejegg> |
civicrm upgraded from dbe3b716 to 68fa85cf |
[production] |
| 18:51 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging2002.codfw.wmnet with OS bullseye |
[production] |
| 18:32 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging2002.codfw.wmnet with reason: host reimage |
[production] |
| 18:28 |
<fab@deploy2002> |
Finished deploy [airflow-dags/research@5edcd7b]: (no justification provided) (duration: 00m 11s) |
[production] |
| 18:27 |
<fab@deploy2002> |
Started deploy [airflow-dags/research@5edcd7b]: (no justification provided) |
[production] |
| 18:27 |
<herron@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging2002.codfw.wmnet with reason: host reimage |
[production] |
| 18:25 |
<hnowlan@deploy2002> |
helmfile [staging] DONE helmfile.d/services/device-analytics: apply |
[production] |
| 18:25 |
<hnowlan@deploy2002> |
helmfile [staging] START helmfile.d/services/device-analytics: apply |
[production] |
| 18:25 |
<hnowlan@deploy2002> |
helmfile [staging] DONE helmfile.d/services/device-analytics: apply |
[production] |
| 18:22 |
<fab@deploy2002> |
Finished deploy [airflow-dags/research@5edcd7b]: (no justification provided) (duration: 00m 30s) |
[production] |
| 18:22 |
<fab@deploy2002> |
Started deploy [airflow-dags/research@5edcd7b]: (no justification provided) |
[production] |
| 18:15 |
<hnowlan@deploy2002> |
helmfile [staging] START helmfile.d/services/device-analytics: apply |
[production] |
| 18:13 |
<brennen@deploy2002> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.40.0-wmf.27 refs T330205 |
[production] |
| 18:13 |
<herron@cumin1001> |
START - Cookbook sre.hosts.reimage for host kafka-logging2002.codfw.wmnet with OS bullseye |
[production] |
| 18:06 |
<hnowlan@deploy2002> |
helmfile [staging] DONE helmfile.d/services/device-analytics: apply |
[production] |
| 18:06 |
<hnowlan@deploy2002> |
helmfile [staging] START helmfile.d/services/device-analytics: apply |
[production] |
| 18:03 |
<brennen> |
1.40.0-wmf.27 train (T330205): no current blockers, rolling to group0. |
[production] |
| 17:59 |
<hnowlan@deploy2002> |
helmfile [codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 17:59 |
<hnowlan@deploy2002> |
helmfile [codfw] START helmfile.d/admin 'apply'. |
[production] |
| 17:58 |
<hnowlan@deploy2002> |
helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 17:56 |
<hnowlan@deploy2002> |
helmfile [staging-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 17:55 |
<hnowlan@deploy2002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 17:55 |
<hnowlan@deploy2002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 17:53 |
<hnowlan@deploy2002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 17:52 |
<hnowlan@deploy2002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 17:52 |
<hnowlan@deploy2002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 17:52 |
<hnowlan@deploy2002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 17:11 |
<aborrero@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudlb2003-dev.codfw.wmnet with OS bullseye |
[production] |
| 17:08 |
<aborrero@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudlb2002-dev.codfw.wmnet with OS bullseye |
[production] |
| 16:49 |
<mvernon@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ms-be2067.codfw.wmnet |
[production] |
| 16:47 |
<sukhe> |
rolling restart of pdns-rec in A:wikidough to pick up config changes |
[production] |
| 16:47 |
<sukhe> |
rolling restart of pdns-rec to pick up config changes |
[production] |
| 16:44 |
<gmodena@deploy1002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply |
[production] |
| 16:44 |
<gmodena@deploy1002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply |
[production] |
| 16:16 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pki2001.codfw.wmnet |
[production] |
| 16:16 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 16:16 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pki2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jbond@cumin1001" |
[production] |
| 16:13 |
<jbond@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: pki2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jbond@cumin1001" |
[production] |
| 16:11 |
<jbond@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:04 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 12:00:00 on cephosd[1001-1005].eqiad.wmnet with reason: Bootstrapping ceph |
[production] |
| 16:04 |
<btullis@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 12:00:00 on cephosd[1001-1005].eqiad.wmnet with reason: Bootstrapping ceph |
[production] |
| 16:00 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts pki2001.codfw.wmnet |
[production] |
| 15:59 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-logging2003.codfw.wmnet with OS bullseye |
[production] |