|
2022-10-04
ยง
|
| 12:26 |
<cgoubert@deploy1002> |
Started deploy [docker-pkg/deploy@24fbee1]: Release 3.0.3 # T310458 |
[production] |
| 12:21 |
<aborrero@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudnet1006.eqiad.wmnet with OS bullseye |
[production] |
| 12:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2181 (re)pooling @ 3%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35340 and previous config saved to /var/cache/conftool/dbconfig/20221004-121917-root.json |
[production] |
| 12:14 |
<volans> |
uploaded python3-gjson_0.1.0 to apt.wikimedia.org bullseye-wikimedia |
[production] |
| 12:13 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1001.eqiad.wmnet with reason: host reimage |
[production] |
| 12:10 |
<aborrero@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudnet1005.eqiad.wmnet with OS bullseye |
[production] |
| 12:09 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1001.eqiad.wmnet with reason: host reimage |
[production] |
| 12:08 |
<hnowlan@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host sessionstore2001.codfw.wmnet with OS buster |
[production] |
| 12:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2181 (re)pooling @ 1%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35339 and previous config saved to /var/cache/conftool/dbconfig/20221004-120413-root.json |
[production] |
| 11:55 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.reimage for host sretest1001.eqiad.wmnet with OS buster |
[production] |
| 11:43 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore2001.codfw.wmnet with reason: host reimage |
[production] |
| 11:40 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore2001.codfw.wmnet with reason: host reimage |
[production] |
| 11:24 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry |
[production] |
| 11:22 |
<jmm@cumin2002> |
START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry |
[production] |
| 11:11 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2181.codfw.wmnet with reason: Upgrading |
[production] |
| 11:10 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2181.codfw.wmnet with reason: Upgrading |
[production] |
| 11:05 |
<jayme> |
published calico 3.23.3 debian packages in bullseye component/calico323 as well as corresponding docker images - T307943 |
[production] |
| 11:04 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudnet1006.eqiad.wmnet with OS bullseye |
[production] |
| 10:58 |
<aborrero@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudnet1006.eqiad.wmnet with OS bullseye |
[production] |
| 10:58 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudnet1006.eqiad.wmnet with OS bullseye |
[production] |
| 10:56 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.reimage for host sessionstore2001.codfw.wmnet with OS buster |
[production] |
| 10:55 |
<aborrero@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudnet1006.eqiad.wmnet with OS bullseye |
[production] |
| 10:54 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudnet1006.eqiad.wmnet with OS bullseye |
[production] |
| 10:54 |
<hnowlan@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sessionstore2001.codfw.wmnet with OS buster |
[production] |
| 10:53 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudnet1005.eqiad.wmnet with OS bullseye |
[production] |
| 10:44 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 135158 |
[production] |
| 10:43 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'configure' for AS: 135158 |
[production] |
| 10:43 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 9119 |
[production] |
| 10:42 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'configure' for AS: 9119 |
[production] |
| 10:41 |
<moritzm> |
installing expat security updates |
[production] |
| 09:59 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.maps.roll-restart (exit_code=1) rolling restart_daemons on A:maps-codfw |
[production] |
| 09:47 |
<btullis@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/eventgate-logging-external: apply |
[production] |
| 09:46 |
<btullis@deploy1002> |
helmfile [eqiad] START helmfile.d/services/eventgate-logging-external: apply |
[production] |
| 09:46 |
<btullis@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/eventgate-logging-external: apply |
[production] |
| 09:46 |
<btullis@deploy1002> |
helmfile [codfw] START helmfile.d/services/eventgate-logging-external: apply |
[production] |
| 09:45 |
<btullis@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/eventgate-logging-external: apply |
[production] |
| 09:44 |
<btullis@deploy1002> |
helmfile [codfw] START helmfile.d/services/eventgate-logging-external: apply |
[production] |
| 09:44 |
<btullis@deploy1002> |
helmfile [staging] DONE helmfile.d/services/eventgate-logging-external: apply |
[production] |
| 09:43 |
<btullis@deploy1002> |
helmfile [staging] START helmfile.d/services/eventgate-logging-external: apply |
[production] |
| 09:42 |
<jayme> |
deployed istio-ingressgateway with additional envoy native metrics to wikikube codfw and eqiad |
[production] |
| 09:40 |
<hnowlan@cumin2002> |
START - Cookbook sre.hosts.reimage for host sessionstore2001.codfw.wmnet with OS buster |
[production] |
| 09:37 |
<jmm@cumin2002> |
START - Cookbook sre.maps.roll-restart rolling restart_daemons on A:maps-codfw |
[production] |
| 09:36 |
<hnowlan@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sessionstore2001.codfw.wmnet with reason: Prep for reimage |
[production] |
| 09:36 |
<hnowlan@cumin2002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on sessionstore2001.codfw.wmnet with reason: Prep for reimage |
[production] |
| 09:36 |
<cgoubert@cumin1001> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 20 hosts |
[production] |
| 09:35 |
<cgoubert@cumin1001> |
START - Cookbook sre.hosts.remove-downtime for 20 hosts |
[production] |
| 09:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2178 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35338 and previous config saved to /var/cache/conftool/dbconfig/20221004-093530-root.json |
[production] |
| 09:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2178 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35337 and previous config saved to /var/cache/conftool/dbconfig/20221004-092025-root.json |
[production] |
| 09:08 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2181.codfw.wmnet with reason: Upgrading |
[production] |
| 09:08 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2181.codfw.wmnet with reason: Upgrading |
[production] |