|
2024-06-18
ยง
|
| 15:39 |
<fabfur> |
upgrade haproxy to v2.8.10 on cp5030,cp5032 (T367756) |
[production] |
| 15:39 |
<fabfur@cumin1002> |
START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp5030.*} and A:cp |
[production] |
| 15:38 |
<fabfur@cumin1002> |
END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp3066.*} and A:cp |
[production] |
| 15:36 |
<fabfur> |
upgrade haproxy to v2.8.10 on cp3066 (T367756) |
[production] |
| 15:35 |
<fabfur@cumin1002> |
START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp3066.*} and A:cp |
[production] |
| 15:35 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'es1040 (re)pooling @ 25%: post T365983 repool', diff saved to https://phabricator.wikimedia.org/P65162 and previous config saved to /var/cache/conftool/dbconfig/20240618-153537-arnaudb.json |
[production] |
| 15:34 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P65161 and previous config saved to /var/cache/conftool/dbconfig/20240618-153430-marostegui.json |
[production] |
| 15:30 |
<mvernon@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host moss-be1003.eqiad.wmnet with OS bookworm |
[production] |
| 15:30 |
<hnowlan@deploy1002> |
helmfile [staging] START helmfile.d/services/sessionstore: sync |
[production] |
| 15:30 |
<hnowlan@deploy1002> |
helmfile [staging] DONE helmfile.d/services/sessionstore: apply |
[production] |
| 15:30 |
<hnowlan@deploy1002> |
helmfile [staging] START helmfile.d/services/sessionstore: apply |
[production] |
| 15:23 |
<isaranto@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . |
[production] |
| 15:20 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'es1040 (re)pooling @ 10%: post T365983 repool', diff saved to https://phabricator.wikimedia.org/P65159 and previous config saved to /var/cache/conftool/dbconfig/20240618-152031-arnaudb.json |
[production] |
| 15:19 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P65158 and previous config saved to /var/cache/conftool/dbconfig/20240618-151923-marostegui.json |
[production] |
| 15:11 |
<mvernon@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on moss-be1003.eqiad.wmnet with reason: host reimage |
[production] |
| 15:07 |
<mvernon@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on moss-be1003.eqiad.wmnet with reason: host reimage |
[production] |
| 15:07 |
<brennen@deploy1002> |
Finished deploy [phabricator/deployment@ef680d8]: revert phab1004 after breakage for T367775 (duration: 00m 15s) |
[production] |
| 15:07 |
<brennen@deploy1002> |
Started deploy [phabricator/deployment@ef680d8]: revert phab1004 after breakage for T367775 |
[production] |
| 15:06 |
<mvernon@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host moss-fe1002.eqiad.wmnet with OS bookworm |
[production] |
| 15:06 |
<bking@deploy1002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow: apply |
[production] |
| 15:06 |
<bking@deploy1002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow: apply |
[production] |
| 15:06 |
<brennen@deploy1002> |
Finished deploy [phabricator/deployment@ebe3a94]: deploy phab1004 for T367775 (duration: 00m 47s) |
[production] |
| 15:05 |
<brennen@deploy1002> |
Started deploy [phabricator/deployment@ebe3a94]: deploy phab1004 for T367775 |
[production] |
| 15:05 |
<brennen@deploy1002> |
Finished deploy [phabricator/deployment@ebe3a94]: deploy phab2002 for T367775 (duration: 00m 36s) |
[production] |
| 15:04 |
<brennen@deploy1002> |
Started deploy [phabricator/deployment@ebe3a94]: deploy phab2002 for T367775 |
[production] |
| 15:04 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1194 (T364069)', diff saved to https://phabricator.wikimedia.org/P65157 and previous config saved to /var/cache/conftool/dbconfig/20240618-150416-marostegui.json |
[production] |
| 15:03 |
<jelto@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2002.codfw.wmnet with reason: Phabricator/Phorge update |
[production] |
| 15:03 |
<jelto@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on phab2002.codfw.wmnet with reason: Phabricator/Phorge update |
[production] |
| 15:03 |
<jelto@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1004.eqiad.wmnet with reason: Phabricator/Phorge update |
[production] |
| 15:03 |
<jelto@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on phab1004.eqiad.wmnet with reason: Phabricator/Phorge update |
[production] |
| 15:00 |
<mforns@deploy1002> |
Finished deploy [airflow-dags/analytics@4f7d29a]: (no justification provided) (duration: 00m 28s) |
[production] |
| 15:00 |
<topranks> |
rebooting lsw1-f7-eqiad to upgrade JunOS on switch T365984 |
[production] |
| 15:00 |
<btullis@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host htmldumper1001.eqiad.wmnet |
[production] |
| 15:00 |
<mforns@deploy1002> |
Started deploy [airflow-dags/analytics@4f7d29a]: (no justification provided) |
[production] |
| 14:57 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:35:00 on an-worker[1172-1174].eqiad.wmnet,es1040.eqiad.wmnet,ms-be1081.eqiad.wmnet with reason: JunOS upgrade lsw1-f7-eqiad |
[production] |
| 14:57 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:35:00 on an-worker[1172-1174].eqiad.wmnet,es1040.eqiad.wmnet,ms-be1081.eqiad.wmnet with reason: JunOS upgrade lsw1-f7-eqiad |
[production] |
| 14:56 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:40:00 on lsw1-f7-eqiad,lsw1-f7-eqiad IPv6,ssw1-e1-eqiad.mgmt,ssw1-f1-eqiad.mgmt with reason: JunOS upgrade lsw1-f7-eqiad |
[production] |
| 14:56 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:40:00 on lsw1-f7-eqiad,lsw1-f7-eqiad IPv6,ssw1-e1-eqiad.mgmt,ssw1-f1-eqiad.mgmt with reason: JunOS upgrade lsw1-f7-eqiad |
[production] |
| 14:53 |
<btullis@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host htmldumper1001.eqiad.wmnet |
[production] |
| 14:49 |
<mvernon@cumin2002> |
START - Cookbook sre.hosts.reimage for host moss-be1003.eqiad.wmnet with OS bookworm |
[production] |
| 14:47 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:40:00 on lsw1-f7-eqiad.mgmt with reason: prep JunOS upgrade lsw1-f7-eqiad |
[production] |
| 14:47 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:40:00 on lsw1-f7-eqiad.mgmt with reason: prep JunOS upgrade lsw1-f7-eqiad |
[production] |
| 14:44 |
<mvernon@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host moss-be1001.eqiad.wmnet with OS bookworm |
[production] |
| 14:44 |
<jynus> |
reenable puppet on backup2002 |
[production] |
| 14:40 |
<klausman@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on ml-serve2001.codfw.wmnet with reason: Hardware maintenance for memory errors |
[production] |
| 14:40 |
<klausman@cumin2002> |
START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on ml-serve2001.codfw.wmnet with reason: Hardware maintenance for memory errors |
[production] |
| 14:39 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'es1040 depool - T365984', diff saved to https://phabricator.wikimedia.org/P65156 and previous config saved to /var/cache/conftool/dbconfig/20240618-143951-arnaudb.json |
[production] |
| 14:39 |
<vgutierrez@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=cp4046.ulsfo.wmnet |
[production] |
| 14:39 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1040.eqiad.wmnet with reason: T365984 |
[production] |
| 14:39 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on es1040.eqiad.wmnet with reason: T365984 |
[production] |