|
2020-07-15
§
|
| 10:50 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 10:45 |
<XioNoX> |
disable ping offload in codfw |
[production] |
| 10:44 |
<XioNoX> |
re-enable ping offload in eqiad |
[production] |
| 10:43 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 10:41 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 10:31 |
<XioNoX> |
disable ping offload in eqiad |
[production] |
| 10:31 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . |
[production] |
| 10:30 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . |
[production] |
| 10:30 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 10:30 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . |
[production] |
| 10:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1120 after reimage', diff saved to https://phabricator.wikimedia.org/P11916 and previous config saved to /var/cache/conftool/dbconfig/20200715-102605-marostegui.json |
[production] |
| 10:20 |
<jayme> |
updating python3-docker-report to 0.0.5-1 on deneb |
[production] |
| 10:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1120 after reimage', diff saved to https://phabricator.wikimedia.org/P11915 and previous config saved to /var/cache/conftool/dbconfig/20200715-100855-marostegui.json |
[production] |
| 10:07 |
<jayme> |
imported docker-report_0.0.5-1 to buster-wikimedia |
[production] |
| 09:47 |
<marostegui> |
Deploy schema change on s8 codfw master, lag will appear on codfw T256685 |
[production] |
| 09:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1120 after reimage', diff saved to https://phabricator.wikimedia.org/P11914 and previous config saved to /var/cache/conftool/dbconfig/20200715-094226-marostegui.json |
[production] |
| 09:22 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 09:21 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 09:19 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . |
[production] |
| 09:19 |
<akosiaris> |
deploy mobileapps in kubernetes to talk HTTPS to the mw API |
[production] |
| 09:10 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . |
[production] |
| 09:10 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . |
[production] |
| 09:07 |
<akosiaris> |
Correction: deploy eventgate-analytics-external in staging, eqiad, codfw for switching to using discovery records and HTTPS for talking to the API |
[production] |
| 09:06 |
<akosiaris> |
deploy eventgate-analytics in staging, eqiad, codfw for switching to using discovery records and HTTPS for talking to the API |
[production] |
| 09:06 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . |
[production] |
| 09:06 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . |
[production] |
| 09:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1099:3318', diff saved to https://phabricator.wikimedia.org/P11913 and previous config saved to /var/cache/conftool/dbconfig/20200715-090545-marostegui.json |
[production] |
| 09:04 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . |
[production] |
| 09:04 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . |
[production] |
| 08:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1120 after reimage', diff saved to https://phabricator.wikimedia.org/P11912 and previous config saved to /var/cache/conftool/dbconfig/20200715-085032-marostegui.json |
[production] |
| 08:39 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:36 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:19 |
<moritzm> |
piwik.wikimedia.org switched to CAS authentication |
[production] |
| 08:19 |
<elukey> |
move piwik.wikimedia.org to CAS (idp.wikimedia.org) |
[production] |
| 07:29 |
<XioNoX> |
delete deprecated AS3209 AMS-IX router |
[production] |
| 06:59 |
<dcausse> |
depooling wdqs1006 (high lag) |
[production] |
| 06:09 |
<marostegui> |
Stop replication on db1120 to avoid having 10.4 -> 10.1 replication for long T254871 |
[production] |
| 06:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1120 for reimage T254871', diff saved to https://phabricator.wikimedia.org/P11911 and previous config saved to /var/cache/conftool/dbconfig/20200715-060649-marostegui.json |
[production] |
| 06:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db1103 to x1 master T254871', diff saved to https://phabricator.wikimedia.org/P11910 and previous config saved to /var/cache/conftool/dbconfig/20200715-060145-marostegui.json |
[production] |
| 06:00 |
<marostegui> |
Starting x1 failover from db1120 to db1103 - T254871 |
[production] |
| 05:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1099:3318 ', diff saved to https://phabricator.wikimedia.org/P11909 and previous config saved to /var/cache/conftool/dbconfig/20200715-052939-marostegui.json |
[production] |
| 04:46 |
<marostegui> |
Start x1 pre failover steps T254871 |
[production] |
| 04:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set db1103 weight to 0 before the switchover T254871', diff saved to https://phabricator.wikimedia.org/P11908 and previous config saved to /var/cache/conftool/dbconfig/20200715-044432-marostegui.json |
[production] |
| 04:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1135', diff saved to https://phabricator.wikimedia.org/P11907 and previous config saved to /var/cache/conftool/dbconfig/20200715-044332-marostegui.json |
[production] |
| 01:45 |
<eileen> |
tools revision changed from a9e7dc1559 to 7b6018a16e |
[production] |
| 00:26 |
<ryankemper@deploy1001> |
Finished deploy [wdqs/wdqs@8f6f660]: 0.3.41 (duration: 15m 10s) |
[production] |
| 00:11 |
<ryankemper@deploy1001> |
Started deploy [wdqs/wdqs@8f6f660]: 0.3.41 |
[production] |