|
2020-07-07
ยง
|
| 09:34 |
<godog> |
bounce logstash on logstash1023 |
[production] |
| 09:33 |
<addshore@deploy1001> |
Synchronized wmf-config/Wikibase.php: [[gerrit:609645]] T257266 T241975 Wikibase: stop using wmgUseEntitySourceBasedFederation (take2) (duration: 00m 59s) |
[production] |
| 09:33 |
<_joe_> |
depooling restbase1025 while we fix the troubled relationship between envoy and proton |
[production] |
| 09:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1096:3315', diff saved to https://phabricator.wikimedia.org/P11773 and previous config saved to /var/cache/conftool/dbconfig/20200707-093345-marostegui.json |
[production] |
| 09:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove weight from es1024 as it is the current master T255755', diff saved to https://phabricator.wikimedia.org/P11772 and previous config saved to /var/cache/conftool/dbconfig/20200707-092635-marostegui.json |
[production] |
| 09:24 |
<James_F> |
1.35.0-wmf.40 was branched at 88ecd6df00a46e432c06c1cf40d5098128abc4d8 for T256668 |
[production] |
| 09:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool es1023 after reimage T255755', diff saved to https://phabricator.wikimedia.org/P11771 and previous config saved to /var/cache/conftool/dbconfig/20200707-092357-marostegui.json |
[production] |
| 09:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es1023 after reimage T255755', diff saved to https://phabricator.wikimedia.org/P11770 and previous config saved to /var/cache/conftool/dbconfig/20200707-091015-marostegui.json |
[production] |
| 08:33 |
<kormat@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es1023 after reimage T255755', diff saved to https://phabricator.wikimedia.org/P11769 and previous config saved to /var/cache/conftool/dbconfig/20200707-083144-marostegui.json |
[production] |
| 08:30 |
<kormat@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:26 |
<XioNoX> |
cr2-codfw> request vmhost snapshot routing-engine both - T257153 |
[production] |
| 08:22 |
<XioNoX> |
cr2-eqsin> request vmhost snapshot - T257153 |
[production] |
| 08:19 |
<XioNoX> |
cr2-eqord> request vmhost snapshot - T257153 |
[production] |
| 08:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es1023 after reimage T255755', diff saved to https://phabricator.wikimedia.org/P11768 and previous config saved to /var/cache/conftool/dbconfig/20200707-081909-marostegui.json |
[production] |
| 08:18 |
<elukey@cumin1001> |
END (ERROR) - Cookbook sre.hadoop.change-distro (exit_code=97) |
[production] |
| 08:17 |
<XioNoX> |
cr2-eqdfw> request vmhost snapshot - T257153 |
[production] |
| 08:15 |
<XioNoX> |
cr3-knams> request vmhost snapshot - T257153 |
[production] |
| 08:15 |
<hashar> |
upgrading and restart CI Jenkins on contint2001 # T256978 |
[production] |
| 08:12 |
<XioNoX> |
cr4-ulsfo> request vmhost snapshot - T257153 |
[production] |
| 08:09 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Depool es2021 for reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11767 and previous config saved to /var/cache/conftool/dbconfig/20200707-080914-kormat.json |
[production] |
| 07:50 |
<marostegui> |
Stop MySQL on db1074 to deploy schema change and remove triggers - T238966 |
[production] |
| 07:45 |
<_joe_> |
restarting restbase again on rb1025 |
[production] |
| 07:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1074 for schema change', diff saved to https://phabricator.wikimedia.org/P11766 and previous config saved to /var/cache/conftool/dbconfig/20200707-074435-marostegui.json |
[production] |
| 07:41 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1079 and db1136 T257216', diff saved to https://phabricator.wikimedia.org/P11765 and previous config saved to /var/cache/conftool/dbconfig/20200707-073918-marostegui.json |
[production] |
| 07:38 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:31 |
<_joe_> |
restarting restbase on restbase1025, reaching proton via envoy for now |
[production] |
| 07:31 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:609644|Revert "Commons: Define entity sources configuration" (T256906, T256907, T256909, T254315, T257266)]] (forgot to git rebase so the last sync was a no-op) (duration: 00m 56s) |
[production] |
| 07:27 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.change-distro |
[production] |
| 07:27 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:609644|Revert "Commons: Define entity sources configuration" (T256906, T256907, T256909, T254315, T257266)]] (duration: 00m 53s) |
[production] |
| 07:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1079 and give more main weight to db1136 T257216', diff saved to https://phabricator.wikimedia.org/P11764 and previous config saved to /var/cache/conftool/dbconfig/20200707-072703-marostegui.json |
[production] |
| 07:24 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/: Config: [[gerrit:609643|Revert "Wikidata client wikis: Define entity sources configuration (take 2)" (T254315, T257266)]] (duration: 00m 56s) |
[production] |
| 07:24 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.stop-cluster (exit_code=0) |
[production] |
| 07:23 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized dblists/wikidataclient.dblist: Config: [[gerrit:609643|Revert "Wikidata client wikis: Define entity sources configuration (take 2)" (T254315, T257266)]] (duration: 00m 56s) |
[production] |
| 07:19 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/Wikibase.php: Config: [[gerrit:609642|Revert "Wikibase: stop using wmgUseEntitySourceBasedFederation" (T241975, T257266)]] (duration: 00m 55s) |
[production] |
| 07:16 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.stop-cluster |
[production] |
| 07:15 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:609641|Revert "Wikibase: Remove config option wmgUseEntitySourceBasedFederation" (T241975, T257266)]] (duration: 00m 57s) |
[production] |
| 07:10 |
<_joe_> |
restart restbase on restbase1025 to pick up the switch to https for cxserver |
[production] |
| 06:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1079 and give more main weight to db1136 T257216', diff saved to https://phabricator.wikimedia.org/P11762 and previous config saved to /var/cache/conftool/dbconfig/20200707-063737-marostegui.json |
[production] |
| 06:29 |
<marostegui> |
Reimage es1023 to Buster T255755 |
[production] |
| 06:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Give db1136 some weight back into main traffic T257216', diff saved to https://phabricator.wikimedia.org/P11761 and previous config saved to /var/cache/conftool/dbconfig/20200707-062008-marostegui.json |
[production] |
| 06:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1079 T257216', diff saved to https://phabricator.wikimedia.org/P11760 and previous config saved to /var/cache/conftool/dbconfig/20200707-061849-marostegui.json |
[production] |
| 05:26 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Enable es5 writes T255755 (duration: 00m 56s) |
[production] |
| 05:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1023 entirely T255755', diff saved to https://phabricator.wikimedia.org/P11759 and previous config saved to /var/cache/conftool/dbconfig/20200707-051620-marostegui.json |
[production] |
| 05:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote es1024 to es5 master T255755', diff saved to https://phabricator.wikimedia.org/P11758 and previous config saved to /var/cache/conftool/dbconfig/20200707-051236-marostegui.json |
[production] |
| 05:02 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Disable es5 writes T255755 (duration: 00m 56s) |
[production] |
| 05:01 |
<marostegui> |
"Starting es failover from es1023 to es1024 - https://phabricator.wikimedia.org/T255755" |
[production] |
| 01:05 |
<ejegg> |
turned on debug logging for Adyen SmashPig |
[production] |
| 00:22 |
<cstone> |
civicrm revision changed from a48caf0f37 to d73ee2e73f |
[production] |