|
2016-04-21
ยง
|
| 15:15 |
<godog> |
[switchover swift #2] upload backends from codfw to eqiad |
[production] |
| 15:09 |
<godog> |
[switchover swift #3] upload codfw to eqiad |
[production] |
| 15:05 |
<godog> |
[switchover swift #4] repool upload eqiad in dns |
[production] |
| 15:04 |
<gehel> |
starting reindex of lost elasticsearch updates during activation of SSL (T132762) |
[production] |
| 15:01 |
<godog> |
[switchover swift #5] upload esams to eqiad |
[production] |
| 14:57 |
<godog> |
[switchover swift #6] upload eqiad to 'direct' |
[production] |
| 14:48 |
<gehel@tin> |
Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 29s) |
[production] |
| 14:45 |
<gehel> |
reverting elasticsearch traffic back to eqiad |
[production] |
| 14:39 |
<jynus> |
enabling replication lag alerts for all dbs |
[production] |
| 14:28 |
<paravoid> |
traffic/mediawiki codfw->eqiad switchover is done |
[production] |
| 14:26 |
<jynus> |
deployed dns updated records for new eqiad masters |
[production] |
| 14:25 |
<_joe_> |
[switchover #8/#3] Running rebuildEntityPerPage.php |
[production] |
| 14:25 |
<_joe_> |
[switchover #8/#2] Enabling crons in eqiad |
[production] |
| 14:24 |
<ori> |
[switchover #8/#1] Starting the jobqueue in eqiad |
[production] |
| 14:23 |
<volans> |
[switchover #8/#3] Re-enable puppet on all eqiad and codfw databases masters |
[production] |
| 14:21 |
<paravoid> |
wikis are read-write again |
[production] |
| 14:21 |
<ori@tin> |
Synchronized wmf-config/db-eqiad.php: [switchover #7/#1] Iac92c8bc6b: Put eqiad in read-write mode for datacenter switchover to eqiad (duration: 00m 39s) |
[production] |
| 14:20 |
<volans> |
[switchover #6/#1] Set database masters RW in eqiad for s1-7, es2-3 and x1 |
[production] |
| 14:19 |
<bblack> |
[switchover #5/#6] Switch Varnish MW backends to eqiad - DONE (confirmed) |
[production] |
| 14:18 |
<Krinkle> |
apache-fast-test warmup finished |
[production] |
| 14:18 |
<godog> |
[switchover #5/#7] finished swift-proxy roll restart |
[production] |
| 14:15 |
<bblack> |
[switchover #5/#6] Switch Varnish MW backends to eqiad - starting |
[production] |
| 14:14 |
<mobrovac> |
[switchover #5/2] restbase {{done}} |
[production] |
| 14:11 |
<subbu> |
parsoid deploy and restarts done |
[production] |
| 14:10 |
<godog> |
[switchover #5/#7] roll-restart swift-proxy in eqiad and codfw |
[production] |
| 14:09 |
<subbu> |
restarting parsoid on all nodes |
[production] |
| 14:09 |
<akosiaris> |
[switchover #5/#3] Misc services cluster (for the action API endpoint) |
[production] |
| 14:09 |
<mobrovac> |
[switchover #5/2] restbase puppet agent -tv && systemctl restart restbase |
[production] |
| 14:08 |
<Krinkle> |
krinkle@tin: bin/apache-fast-test wiki-urls-warmup1000.txt eqiad |
[production] |
| 14:08 |
<subbu> |
syncing parsoid code |
[production] |
| 14:07 |
<volans> |
[switchover #5/#5] Switch parsercache RO/RW |
[production] |
| 14:07 |
<_joe_> |
[switchover #5/1] switching redis replication manually |
[production] |
| 14:07 |
<ori@tin> |
Synchronized wmf-config/CommonSettings.php: [switchover #4/#2] I0e85c3d20: Switch wmfMasterDatacenter to eqiad (duration: 00m 26s) |
[production] |
| 14:06 |
<_joe_> |
[switchover #4/1] puppet merged |
[production] |
| 14:04 |
<Krinkle> |
krinkle@tin: bin/apache-fast-test wiki-urls-warmup1000.txt eqiad |
[production] |
| 14:03 |
<volans> |
[switchover #3/#1] Set active site's databases (masters) in read-only mode except parsercache ones. |
[production] |
| 14:03 |
<_joe_> |
[swichover 3/2 wipe memcacheds] |
[production] |
| 14:02 |
<paravoid> |
wikis now in planned read-only mode, cf. http://blog.wikimedia.org/2016/04/18/wikimedia-server-switch/ |
[production] |
| 14:01 |
<ori@tin> |
Synchronized wmf-config/db-codfw.php: [switchover #2/#1] Id8b2e7a05: Set codfw databases to read-only mode (duration: 00m 24s) |
[production] |
| 14:00 |
<jynus> |
disabled all db lag alerts |
[production] |
| 13:55 |
<volans> |
[switchover #1/#6] Switch pt-heartbeat from active site (codfw) to new site (eqiad) masters |
[production] |
| 13:54 |
<ori> |
[switchover #1/2] stopping jobrunners in codfw |
[production] |
| 13:54 |
<_joe_> |
[switchover #1/3] stopping crons on wasat |
[production] |
| 13:52 |
<volans> |
[switchover #1/#5] Set final $master status for databases in advance |
[production] |
| 13:50 |
<volans> |
[switchover #1/#4] Disable puppet on all eqiad and codfw databases masters |
[production] |
| 13:50 |
<paravoid> |
commencing codfw->eqiad datacenter switchover |
[production] |
| 13:39 |
<ori@tin> |
Synchronized wmf-config/InitialiseSettings.php: I2171f6b1: Enable MessageCacheError log channel (duration: 00m 25s) |
[production] |
| 13:37 |
<bblack> |
[traffic codfw switch revert #3] - DNS TTL done, bulk of end-user traffic rebalanced, graphs starting to level off at new normals, as done as it gets from our end |
[production] |
| 13:31 |
<bblack> |
[traffic codfw switch revert #4] - done & confirmed |
[production] |
| 13:28 |
<bblack> |
[traffic codfw switch revert #4] - merge -> start salted puppet |
[production] |