|
2020-07-07
ยง
|
| 12:57 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:44 |
<kormat> |
starting (codfw) es5 failover from es2020 to es2021 T257284 |
[production] |
| 12:30 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Set es2021 to weight 50 T257284', diff saved to https://phabricator.wikimedia.org/P11787 and previous config saved to /var/cache/conftool/dbconfig/20200707-123003-kormat.json |
[production] |
| 12:12 |
<jforrester@deploy1001> |
Finished scap: Full scap and testwikis to 1.35.0-wmf.40 for T256668 (duration: 33m 09s) |
[production] |
| 12:01 |
<marostegui> |
Deploy schema change on labswiki (wikitech) master - T253276 |
[production] |
| 11:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1082', diff saved to https://phabricator.wikimedia.org/P11786 and previous config saved to /var/cache/conftool/dbconfig/20200707-115838-marostegui.json |
[production] |
| 11:39 |
<jforrester@deploy1001> |
Started scap: Full scap and testwikis to 1.35.0-wmf.40 for T256668 |
[production] |
| 11:38 |
<jforrester@deploy1001> |
scap failed: LockFailedError Failed to acquire lock "/var/lock/scap.operations_mediawiki-config.lock"; owner is "jforrester"; reason is "testwikis wikis to 1.35.0-wmf.40" (duration: 00m 00s) |
[production] |
| 11:33 |
<moritzm> |
installing PHP 7.0 security updates |
[production] |
| 11:29 |
<marostegui> |
Deploy schema change on db1082, this will create lag on s5 labs |
[production] |
| 11:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1082', diff saved to https://phabricator.wikimedia.org/P11784 and previous config saved to /var/cache/conftool/dbconfig/20200707-112926-marostegui.json |
[production] |
| 11:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1130', diff saved to https://phabricator.wikimedia.org/P11783 and previous config saved to /var/cache/conftool/dbconfig/20200707-112830-marostegui.json |
[production] |
| 11:26 |
<godog> |
test bumping logstash7 batch size to 256 |
[production] |
| 11:17 |
<moritzm> |
prune PHP 7.0 packages from mwdebug1001/2001/2002 |
[production] |
| 11:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1130', diff saved to https://phabricator.wikimedia.org/P11782 and previous config saved to /var/cache/conftool/dbconfig/20200707-110506-marostegui.json |
[production] |
| 11:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1110', diff saved to https://phabricator.wikimedia.org/P11781 and previous config saved to /var/cache/conftool/dbconfig/20200707-110412-marostegui.json |
[production] |
| 10:57 |
<moritzm> |
prune PHP 7.0 packages from mw2190-mw2214 |
[production] |
| 10:46 |
<jforrester@deploy1001> |
Started scap: testwikis wikis to 1.35.0-wmf.40 |
[production] |
| 10:44 |
<jforrester@deploy1001> |
Pruned MediaWiki: 1.35.0-wmf.38 (duration: 17m 23s) |
[production] |
| 10:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1110', diff saved to https://phabricator.wikimedia.org/P11780 and previous config saved to /var/cache/conftool/dbconfig/20200707-103255-marostegui.json |
[production] |
| 10:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1113:3315', diff saved to https://phabricator.wikimedia.org/P11779 and previous config saved to /var/cache/conftool/dbconfig/20200707-102757-marostegui.json |
[production] |
| 10:26 |
<moritzm> |
prune PHP 7.0 packages from mw2135-mw2147 |
[production] |
| 10:12 |
<addshore@deploy1001> |
Synchronized wmf-config/config/testcommonswiki.yaml: [[gerrit:609985]] Make testcommonswiki a testwikidata client T257266 PT2/2 (duration: 00m 55s) |
[production] |
| 10:11 |
<addshore@deploy1001> |
sync-file aborted: [[gerrit:609985]] Make testcommonswiki a testwikidata client T257266 PT1/2 (duration: 00m 00s) |
[production] |
| 10:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1113:3315', diff saved to https://phabricator.wikimedia.org/P11778 and previous config saved to /var/cache/conftool/dbconfig/20200707-101043-marostegui.json |
[production] |
| 10:10 |
<addshore@deploy1001> |
Synchronized dblists/wikidataclient-test.dblist: [[gerrit:609985]] Make testcommonswiki a testwikidata client T257266 PT1/2 (duration: 00m 56s) |
[production] |
| 10:08 |
<addshore@deploy1001> |
sync-file aborted: [[gerrit:609985]] Make testcommonswiki a testwikidata client T257266 PT1/2 (duration: 00m 36s) |
[production] |
| 10:06 |
<elukey> |
decommission archiva1001 |
[production] |
| 10:05 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 10:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1144:3315', diff saved to https://phabricator.wikimedia.org/P11777 and previous config saved to /var/cache/conftool/dbconfig/20200707-100328-marostegui.json |
[production] |
| 10:03 |
<elukey@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 10:03 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:03 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 09:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1144:3315', diff saved to https://phabricator.wikimedia.org/P11776 and previous config saved to /var/cache/conftool/dbconfig/20200707-095443-marostegui.json |
[production] |
| 09:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1096:3315', diff saved to https://phabricator.wikimedia.org/P11775 and previous config saved to /var/cache/conftool/dbconfig/20200707-095428-marostegui.json |
[production] |
| 09:42 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:609971]] T257266 Enable sitelinks to testcommons from test wikidata sites (duration: 00m 56s) |
[production] |
| 09:40 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Repool es2021 after reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11774 and previous config saved to /var/cache/conftool/dbconfig/20200707-094017-kormat.json |
[production] |
| 09:37 |
<addshore@deploy1001> |
Synchronized wmf-config: [[gerrit:609986]] T257266 T241975 Wikibase: Remove config option wmgUseEntitySourceBasedFederation (take2) (duration: 00m 57s) |
[production] |
| 09:36 |
<_joe_> |
errata: restbase2010, not 2009 |
[production] |
| 09:36 |
<_joe_> |
applying the new configuration using the service proxy to restbase2009 too |
[production] |
| 09:34 |
<godog> |
bounce logstash on logstash1023 |
[production] |
| 09:33 |
<addshore@deploy1001> |
Synchronized wmf-config/Wikibase.php: [[gerrit:609645]] T257266 T241975 Wikibase: stop using wmgUseEntitySourceBasedFederation (take2) (duration: 00m 59s) |
[production] |
| 09:33 |
<_joe_> |
depooling restbase1025 while we fix the troubled relationship between envoy and proton |
[production] |
| 09:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1096:3315', diff saved to https://phabricator.wikimedia.org/P11773 and previous config saved to /var/cache/conftool/dbconfig/20200707-093345-marostegui.json |
[production] |
| 09:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove weight from es1024 as it is the current master T255755', diff saved to https://phabricator.wikimedia.org/P11772 and previous config saved to /var/cache/conftool/dbconfig/20200707-092635-marostegui.json |
[production] |
| 09:24 |
<James_F> |
1.35.0-wmf.40 was branched at 88ecd6df00a46e432c06c1cf40d5098128abc4d8 for T256668 |
[production] |
| 09:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool es1023 after reimage T255755', diff saved to https://phabricator.wikimedia.org/P11771 and previous config saved to /var/cache/conftool/dbconfig/20200707-092357-marostegui.json |
[production] |
| 09:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es1023 after reimage T255755', diff saved to https://phabricator.wikimedia.org/P11770 and previous config saved to /var/cache/conftool/dbconfig/20200707-091015-marostegui.json |
[production] |
| 08:33 |
<kormat@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es1023 after reimage T255755', diff saved to https://phabricator.wikimedia.org/P11769 and previous config saved to /var/cache/conftool/dbconfig/20200707-083144-marostegui.json |
[production] |