|
2022-10-04
ยง
|
| 13:13 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. |
[production] |
| 13:13 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 13:13 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 13:12 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 13:11 |
<awight@deploy1002> |
awight and stang: Backport for [[gerrit:837756|throttle: Add throttle rule for 2022-10-13 (T319244)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet |
[production] |
| 13:08 |
<awight@deploy1002> |
Started scap: Backport for [[gerrit:837756|throttle: Add throttle rule for 2022-10-13 (T319244)]] |
[production] |
| 13:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2181 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35343 and previous config saved to /var/cache/conftool/dbconfig/20221004-130432-root.json |
[production] |
| 12:58 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudnet1006.eqiad.wmnet with reason: host reimage |
[production] |
| 12:56 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudnet1005.eqiad.wmnet with reason: host reimage |
[production] |
| 12:53 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudnet1006.eqiad.wmnet with reason: host reimage |
[production] |
| 12:53 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudnet1005.eqiad.wmnet with reason: host reimage |
[production] |
| 12:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2181 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35342 and previous config saved to /var/cache/conftool/dbconfig/20221004-124927-root.json |
[production] |
| 12:37 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudnet1006.eqiad.wmnet with OS bullseye |
[production] |
| 12:37 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudnet1005.eqiad.wmnet with OS bullseye |
[production] |
| 12:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2181 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35341 and previous config saved to /var/cache/conftool/dbconfig/20221004-123422-root.json |
[production] |
| 12:31 |
<cgoubert@deploy1002> |
Finished deploy [docker-pkg/deploy@24fbee1]: Release 3.0.3 # T310458 (duration: 00m 58s) |
[production] |
| 12:30 |
<cgoubert@deploy1002> |
Started deploy [docker-pkg/deploy@24fbee1]: Release 3.0.3 # T310458 |
[production] |
| 12:29 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1001.eqiad.wmnet with OS buster |
[production] |
| 12:26 |
<cgoubert@deploy1002> |
Finished deploy [docker-pkg/deploy@24fbee1]: Release 3.0.3 # T310458 (duration: 00m 14s) |
[production] |
| 12:26 |
<cgoubert@deploy1002> |
Started deploy [docker-pkg/deploy@24fbee1]: Release 3.0.3 # T310458 |
[production] |
| 12:21 |
<aborrero@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudnet1006.eqiad.wmnet with OS bullseye |
[production] |
| 12:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2181 (re)pooling @ 3%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35340 and previous config saved to /var/cache/conftool/dbconfig/20221004-121917-root.json |
[production] |
| 12:14 |
<volans> |
uploaded python3-gjson_0.1.0 to apt.wikimedia.org bullseye-wikimedia |
[production] |
| 12:13 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1001.eqiad.wmnet with reason: host reimage |
[production] |
| 12:10 |
<aborrero@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudnet1005.eqiad.wmnet with OS bullseye |
[production] |
| 12:09 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1001.eqiad.wmnet with reason: host reimage |
[production] |
| 12:08 |
<hnowlan@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host sessionstore2001.codfw.wmnet with OS buster |
[production] |
| 12:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2181 (re)pooling @ 1%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35339 and previous config saved to /var/cache/conftool/dbconfig/20221004-120413-root.json |
[production] |
| 11:55 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.reimage for host sretest1001.eqiad.wmnet with OS buster |
[production] |
| 11:43 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore2001.codfw.wmnet with reason: host reimage |
[production] |
| 11:40 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore2001.codfw.wmnet with reason: host reimage |
[production] |
| 11:24 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry |
[production] |
| 11:22 |
<jmm@cumin2002> |
START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry |
[production] |
| 11:11 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2181.codfw.wmnet with reason: Upgrading |
[production] |
| 11:10 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2181.codfw.wmnet with reason: Upgrading |
[production] |
| 11:05 |
<jayme> |
published calico 3.23.3 debian packages in bullseye component/calico323 as well as corresponding docker images - T307943 |
[production] |
| 11:04 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudnet1006.eqiad.wmnet with OS bullseye |
[production] |
| 10:58 |
<aborrero@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudnet1006.eqiad.wmnet with OS bullseye |
[production] |
| 10:58 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudnet1006.eqiad.wmnet with OS bullseye |
[production] |
| 10:56 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.reimage for host sessionstore2001.codfw.wmnet with OS buster |
[production] |
| 10:55 |
<aborrero@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudnet1006.eqiad.wmnet with OS bullseye |
[production] |
| 10:54 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudnet1006.eqiad.wmnet with OS bullseye |
[production] |
| 10:54 |
<hnowlan@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sessionstore2001.codfw.wmnet with OS buster |
[production] |
| 10:53 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudnet1005.eqiad.wmnet with OS bullseye |
[production] |
| 10:44 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 135158 |
[production] |
| 10:43 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'configure' for AS: 135158 |
[production] |
| 10:43 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 9119 |
[production] |
| 10:42 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'configure' for AS: 9119 |
[production] |
| 10:41 |
<moritzm> |
installing expat security updates |
[production] |
| 09:59 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.maps.roll-restart (exit_code=1) rolling restart_daemons on A:maps-codfw |
[production] |