|
2022-10-04
ยง
|
| 09:59 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.maps.roll-restart (exit_code=1) rolling restart_daemons on A:maps-codfw |
[production] |
| 09:47 |
<btullis@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/eventgate-logging-external: apply |
[production] |
| 09:46 |
<btullis@deploy1002> |
helmfile [eqiad] START helmfile.d/services/eventgate-logging-external: apply |
[production] |
| 09:46 |
<btullis@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/eventgate-logging-external: apply |
[production] |
| 09:46 |
<btullis@deploy1002> |
helmfile [codfw] START helmfile.d/services/eventgate-logging-external: apply |
[production] |
| 09:45 |
<btullis@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/eventgate-logging-external: apply |
[production] |
| 09:44 |
<btullis@deploy1002> |
helmfile [codfw] START helmfile.d/services/eventgate-logging-external: apply |
[production] |
| 09:44 |
<btullis@deploy1002> |
helmfile [staging] DONE helmfile.d/services/eventgate-logging-external: apply |
[production] |
| 09:43 |
<btullis@deploy1002> |
helmfile [staging] START helmfile.d/services/eventgate-logging-external: apply |
[production] |
| 09:42 |
<jayme> |
deployed istio-ingressgateway with additional envoy native metrics to wikikube codfw and eqiad |
[production] |
| 09:40 |
<hnowlan@cumin2002> |
START - Cookbook sre.hosts.reimage for host sessionstore2001.codfw.wmnet with OS buster |
[production] |
| 09:37 |
<jmm@cumin2002> |
START - Cookbook sre.maps.roll-restart rolling restart_daemons on A:maps-codfw |
[production] |
| 09:36 |
<hnowlan@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sessionstore2001.codfw.wmnet with reason: Prep for reimage |
[production] |
| 09:36 |
<hnowlan@cumin2002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on sessionstore2001.codfw.wmnet with reason: Prep for reimage |
[production] |
| 09:36 |
<cgoubert@cumin1001> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 20 hosts |
[production] |
| 09:35 |
<cgoubert@cumin1001> |
START - Cookbook sre.hosts.remove-downtime for 20 hosts |
[production] |
| 09:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2178 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35338 and previous config saved to /var/cache/conftool/dbconfig/20221004-093530-root.json |
[production] |
| 09:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2178 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35337 and previous config saved to /var/cache/conftool/dbconfig/20221004-092025-root.json |
[production] |
| 09:08 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2181.codfw.wmnet with reason: Upgrading |
[production] |
| 09:08 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2181.codfw.wmnet with reason: Upgrading |
[production] |
| 09:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2178 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35336 and previous config saved to /var/cache/conftool/dbconfig/20221004-090520-root.json |
[production] |
| 08:56 |
<cgoubert@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 20 hosts with reason: php7.2 removal |
[production] |
| 08:55 |
<cgoubert@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 20 hosts with reason: php7.2 removal |
[production] |
| 08:52 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2181.codfw.wmnet with reason: Upgrading |
[production] |
| 08:52 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2181.codfw.wmnet with reason: Upgrading |
[production] |
| 08:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2178 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35335 and previous config saved to /var/cache/conftool/dbconfig/20221004-085015-root.json |
[production] |
| 08:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2178 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35334 and previous config saved to /var/cache/conftool/dbconfig/20221004-083511-root.json |
[production] |
| 08:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2178 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35333 and previous config saved to /var/cache/conftool/dbconfig/20221004-082005-root.json |
[production] |
| 08:17 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2181.codfw.wmnet with reason: Upgrading |
[production] |
| 08:16 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2181.codfw.wmnet with reason: Upgrading |
[production] |
| 08:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2178 (re)pooling @ 3%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35332 and previous config saved to /var/cache/conftool/dbconfig/20221004-080500-root.json |
[production] |
| 08:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2181', diff saved to https://phabricator.wikimedia.org/P35331 and previous config saved to /var/cache/conftool/dbconfig/20221004-080338-root.json |
[production] |
| 07:52 |
<moritzm> |
installing libdatetime-timezone-perl updates (catching up with latest timezone changes) |
[production] |
| 07:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2178 (re)pooling @ 1%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35330 and previous config saved to /var/cache/conftool/dbconfig/20221004-074955-root.json |
[production] |
| 07:36 |
<elukey@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/eventgate-logging-external: sync |
[production] |
| 07:36 |
<elukey@deploy1002> |
helmfile [codfw] START helmfile.d/services/eventgate-logging-external: sync |
[production] |
| 07:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1189 (re)pooling @ 100%: After HW maintenance', diff saved to https://phabricator.wikimedia.org/P35329 and previous config saved to /var/cache/conftool/dbconfig/20221004-072158-root.json |
[production] |
| 07:16 |
<elukey> |
restart kafka on kafka-logging1001 to pick up its new PKI TLS cert |
[production] |
| 07:11 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:20:00 on kafka-logging1001.eqiad.wmnet with reason: Kafka PKI upgrade |
[production] |
| 07:11 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 0:20:00 on kafka-logging1001.eqiad.wmnet with reason: Kafka PKI upgrade |
[production] |
| 07:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1189 (re)pooling @ 75%: After HW maintenance', diff saved to https://phabricator.wikimedia.org/P35328 and previous config saved to /var/cache/conftool/dbconfig/20221004-070653-root.json |
[production] |
| 06:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1189 (re)pooling @ 50%: After HW maintenance', diff saved to https://phabricator.wikimedia.org/P35327 and previous config saved to /var/cache/conftool/dbconfig/20221004-065148-root.json |
[production] |
| 06:43 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 06:42 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 06:42 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 06:39 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 06:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1189 (re)pooling @ 25%: After HW maintenance', diff saved to https://phabricator.wikimedia.org/P35326 and previous config saved to /var/cache/conftool/dbconfig/20221004-063643-root.json |
[production] |
| 06:33 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 25885 |
[production] |
| 06:32 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'configure' for AS: 25885 |
[production] |
| 06:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1189 (re)pooling @ 10%: After HW maintenance', diff saved to https://phabricator.wikimedia.org/P35325 and previous config saved to /var/cache/conftool/dbconfig/20221004-062138-root.json |
[production] |