|
2024-07-09
ยง
|
| 16:19 |
<btullis@deploy1002> |
Finished deploy [analytics/refinery@a203f30]: Regular analytics weekly train [analytics/refinery@a203f30c] (duration: 04m 51s) |
[production] |
| 16:14 |
<btullis@deploy1002> |
Started deploy [analytics/refinery@a203f30]: Regular analytics weekly train [analytics/refinery@a203f30c] |
[production] |
| 16:14 |
<btullis@deploy1002> |
Finished deploy [analytics/refinery@a203f30]: Regular analytics weekly train [analytics/refinery@a203f30c] (duration: 09m 23s) |
[production] |
| 16:14 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1199 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P66071 and previous config saved to /var/cache/conftool/dbconfig/20240709-161404-root.json |
[production] |
| 16:14 |
<btullis> |
pooled druid1010 |
[production] |
| 16:13 |
<btullis> |
unset noout mode on the cephosd cluster |
[production] |
| 16:13 |
<btullis> |
uncordoned dse-k8s-worker1006 |
[production] |
| 16:12 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1198 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P66070 and previous config saved to /var/cache/conftool/dbconfig/20240709-161230-root.json |
[production] |
| 16:12 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1192 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P66069 and previous config saved to /var/cache/conftool/dbconfig/20240709-161222-root.json |
[production] |
| 16:07 |
<hnowlan@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply |
[production] |
| 16:04 |
<btullis@deploy1002> |
Started deploy [analytics/refinery@a203f30]: Regular analytics weekly train [analytics/refinery@a203f30c] |
[production] |
| 15:58 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1199 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P66068 and previous config saved to /var/cache/conftool/dbconfig/20240709-155858-root.json |
[production] |
| 15:57 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1198 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P66067 and previous config saved to /var/cache/conftool/dbconfig/20240709-155724-root.json |
[production] |
| 15:57 |
<hnowlan@deploy1002> |
helmfile [codfw] START helmfile.d/services/shellbox-video: apply |
[production] |
| 15:57 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1192 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P66066 and previous config saved to /var/cache/conftool/dbconfig/20240709-155717-root.json |
[production] |
| 15:56 |
<hnowlan@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply |
[production] |
| 15:46 |
<hnowlan@deploy1002> |
helmfile [codfw] START helmfile.d/services/shellbox-video: apply |
[production] |
| 15:44 |
<hnowlan@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply |
[production] |
| 15:44 |
<ayounsi@cumin1002> |
END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.6 to netbox-next - ayounsi@cumin1002 - T336275 |
[production] |
| 15:44 |
<hnowlan@deploy1002> |
helmfile [eqiad] START helmfile.d/services/shellbox-video: apply |
[production] |
| 15:43 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1199 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P66065 and previous config saved to /var/cache/conftool/dbconfig/20240709-154353-root.json |
[production] |
| 15:42 |
<ayounsi@cumin1002> |
START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.6 to netbox-next - ayounsi@cumin1002 - T336275 |
[production] |
| 15:42 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1198 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P66064 and previous config saved to /var/cache/conftool/dbconfig/20240709-154219-root.json |
[production] |
| 15:42 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1192 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P66063 and previous config saved to /var/cache/conftool/dbconfig/20240709-154211-root.json |
[production] |
| 15:41 |
<swfrench@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/commons-impact-analytics: apply |
[production] |
| 15:41 |
<swfrench@deploy1002> |
helmfile [eqiad] START helmfile.d/services/commons-impact-analytics: apply |
[production] |
| 15:39 |
<swfrench@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/commons-impact-analytics: apply |
[production] |
| 15:38 |
<swfrench@deploy1002> |
helmfile [codfw] START helmfile.d/services/commons-impact-analytics: apply |
[production] |
| 15:35 |
<sukhe> |
remove traffic-dnsbox VM on cloud-vps: T360710 |
[production] |
| 15:28 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1199 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P66062 and previous config saved to /var/cache/conftool/dbconfig/20240709-152847-root.json |
[production] |
| 15:27 |
<hnowlan@cumin1002> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 9 hosts |
[production] |
| 15:27 |
<hnowlan@cumin1002> |
START - Cookbook sre.hosts.remove-downtime for 9 hosts |
[production] |
| 15:27 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1198 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P66061 and previous config saved to /var/cache/conftool/dbconfig/20240709-152713-root.json |
[production] |
| 15:27 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1192 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P66060 and previous config saved to /var/cache/conftool/dbconfig/20240709-152706-root.json |
[production] |
| 15:14 |
<ayounsi@cumin1002> |
END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.6 to netbox-next - ayounsi@cumin1002 - T336275 |
[production] |
| 15:12 |
<ayounsi@cumin1002> |
START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.6 to netbox-next - ayounsi@cumin1002 - T336275 |
[production] |
| 15:11 |
<ayounsi@cumin1002> |
END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.6 to netbox-next - ayounsi@cumin1002 - T336275 |
[production] |
| 15:08 |
<ayounsi@cumin1002> |
START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.6 to netbox-next - ayounsi@cumin1002 - T336275 |
[production] |
| 15:04 |
<topranks> |
rebooting lsw1-e3-eqiad to install updated JunOS version T365998 |
[production] |
| 15:03 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:40:00 on 27 hosts with reason: JunOS upgrade lsw1-e3-eqiad |
[production] |
| 15:02 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:40:00 on 27 hosts with reason: JunOS upgrade lsw1-e3-eqiad |
[production] |
| 15:01 |
<hnowlan@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 9 hosts with reason: network maintenance |
[production] |
| 15:01 |
<hnowlan@cumin1002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on 9 hosts with reason: network maintenance |
[production] |
| 15:00 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on lsw1-e3-eqiad,lsw1-e3-eqiad IPv6,ssw1-e1-eqiad.mgmt,ssw1-f1-eqiad.mgmt with reason: JunOS upgrade lsw1-e3-eqiad |
[production] |
| 14:59 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on lsw1-e3-eqiad,lsw1-e3-eqiad IPv6,ssw1-e1-eqiad.mgmt,ssw1-f1-eqiad.mgmt with reason: JunOS upgrade lsw1-e3-eqiad |
[production] |
| 14:54 |
<swfrench@deploy1002> |
helmfile [staging] DONE helmfile.d/services/commons-impact-analytics: apply |
[production] |
| 14:53 |
<swfrench@deploy1002> |
helmfile [staging] START helmfile.d/services/commons-impact-analytics: apply |
[production] |
| 14:53 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:40:00 on lsw1-e3-eqiad.mgmt with reason: prep JunOS upgrade lsw1-e3-eqiad |
[production] |
| 14:53 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:40:00 on lsw1-e3-eqiad.mgmt with reason: prep JunOS upgrade lsw1-e3-eqiad |
[production] |
| 14:50 |
<hashar> |
Restart Gerrit primary on gerrit1003 to apply a configuration change | T367505 |
[production] |