|
2016-04-21
ยง
|
| 14:39 |
<jynus> |
enabling replication lag alerts for all dbs |
[production] |
| 14:28 |
<paravoid> |
traffic/mediawiki codfw->eqiad switchover is done |
[production] |
| 14:26 |
<jynus> |
deployed dns updated records for new eqiad masters |
[production] |
| 14:25 |
<_joe_> |
[switchover #8/#3] Running rebuildEntityPerPage.php |
[production] |
| 14:25 |
<_joe_> |
[switchover #8/#2] Enabling crons in eqiad |
[production] |
| 14:24 |
<ori> |
[switchover #8/#1] Starting the jobqueue in eqiad |
[production] |
| 14:23 |
<volans> |
[switchover #8/#3] Re-enable puppet on all eqiad and codfw databases masters |
[production] |
| 14:21 |
<paravoid> |
wikis are read-write again |
[production] |
| 14:21 |
<ori@tin> |
Synchronized wmf-config/db-eqiad.php: [switchover #7/#1] Iac92c8bc6b: Put eqiad in read-write mode for datacenter switchover to eqiad (duration: 00m 39s) |
[production] |
| 14:20 |
<volans> |
[switchover #6/#1] Set database masters RW in eqiad for s1-7, es2-3 and x1 |
[production] |
| 14:19 |
<bblack> |
[switchover #5/#6] Switch Varnish MW backends to eqiad - DONE (confirmed) |
[production] |
| 14:18 |
<Krinkle> |
apache-fast-test warmup finished |
[production] |
| 14:18 |
<godog> |
[switchover #5/#7] finished swift-proxy roll restart |
[production] |
| 14:15 |
<bblack> |
[switchover #5/#6] Switch Varnish MW backends to eqiad - starting |
[production] |
| 14:14 |
<mobrovac> |
[switchover #5/2] restbase {{done}} |
[production] |
| 14:11 |
<subbu> |
parsoid deploy and restarts done |
[production] |
| 14:10 |
<godog> |
[switchover #5/#7] roll-restart swift-proxy in eqiad and codfw |
[production] |
| 14:09 |
<subbu> |
restarting parsoid on all nodes |
[production] |
| 14:09 |
<akosiaris> |
[switchover #5/#3] Misc services cluster (for the action API endpoint) |
[production] |
| 14:09 |
<mobrovac> |
[switchover #5/2] restbase puppet agent -tv && systemctl restart restbase |
[production] |
| 14:08 |
<Krinkle> |
krinkle@tin: bin/apache-fast-test wiki-urls-warmup1000.txt eqiad |
[production] |
| 14:08 |
<subbu> |
syncing parsoid code |
[production] |
| 14:07 |
<volans> |
[switchover #5/#5] Switch parsercache RO/RW |
[production] |
| 14:07 |
<_joe_> |
[switchover #5/1] switching redis replication manually |
[production] |
| 14:07 |
<ori@tin> |
Synchronized wmf-config/CommonSettings.php: [switchover #4/#2] I0e85c3d20: Switch wmfMasterDatacenter to eqiad (duration: 00m 26s) |
[production] |
| 14:06 |
<_joe_> |
[switchover #4/1] puppet merged |
[production] |
| 14:04 |
<Krinkle> |
krinkle@tin: bin/apache-fast-test wiki-urls-warmup1000.txt eqiad |
[production] |
| 14:03 |
<volans> |
[switchover #3/#1] Set active site's databases (masters) in read-only mode except parsercache ones. |
[production] |
| 14:03 |
<_joe_> |
[swichover 3/2 wipe memcacheds] |
[production] |
| 14:02 |
<paravoid> |
wikis now in planned read-only mode, cf. http://blog.wikimedia.org/2016/04/18/wikimedia-server-switch/ |
[production] |
| 14:01 |
<ori@tin> |
Synchronized wmf-config/db-codfw.php: [switchover #2/#1] Id8b2e7a05: Set codfw databases to read-only mode (duration: 00m 24s) |
[production] |
| 14:00 |
<jynus> |
disabled all db lag alerts |
[production] |
| 13:55 |
<volans> |
[switchover #1/#6] Switch pt-heartbeat from active site (codfw) to new site (eqiad) masters |
[production] |
| 13:54 |
<ori> |
[switchover #1/2] stopping jobrunners in codfw |
[production] |
| 13:54 |
<_joe_> |
[switchover #1/3] stopping crons on wasat |
[production] |
| 13:52 |
<volans> |
[switchover #1/#5] Set final $master status for databases in advance |
[production] |
| 13:50 |
<volans> |
[switchover #1/#4] Disable puppet on all eqiad and codfw databases masters |
[production] |
| 13:50 |
<paravoid> |
commencing codfw->eqiad datacenter switchover |
[production] |
| 13:39 |
<ori@tin> |
Synchronized wmf-config/InitialiseSettings.php: I2171f6b1: Enable MessageCacheError log channel (duration: 00m 25s) |
[production] |
| 13:37 |
<bblack> |
[traffic codfw switch revert #3] - DNS TTL done, bulk of end-user traffic rebalanced, graphs starting to level off at new normals, as done as it gets from our end |
[production] |
| 13:31 |
<bblack> |
[traffic codfw switch revert #4] - done & confirmed |
[production] |
| 13:28 |
<bblack> |
[traffic codfw switch revert #4] - merge -> start salted puppet |
[production] |
| 13:27 |
<bblack> |
[traffic codfw switch revert #2] - done & confirmed |
[production] |
| 13:25 |
<bblack> |
[traffic codfw switch revert #3] - merge -> authdns-update |
[production] |
| 13:24 |
<bblack> |
[traffic codfw switch revert #2] - merge -> start salted puppet |
[production] |
| 13:23 |
<bblack> |
[traffic codfw switch revert #1] - done & confirmed |
[production] |
| 13:23 |
<bblack> |
[traffic codfw switch revert #1] - merge -> start salted puppet (@13:20, late log) |
[production] |
| 13:21 |
<ori@tin> |
Synchronized php-1.27.0-wmf.21/includes: Ie9799f5ea: Make MessageCache handle lock timeouts better (duration: 01m 18s) |
[production] |
| 13:12 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Temporarely increase es1* master weight to add connection capacity (duration: 00m 37s) |
[production] |
| 09:57 |
<elukey> |
removed apache2 logrotate config manually from argon as temp patch to remove cronspam from root@ (T132896) |
[production] |