|
2016-04-21
§
|
| 15:34 |
<_joe_> |
services switchover done |
[production] |
| 15:23 |
<filippo@tin> |
Synchronized wmf-config/filebackend-production.php: [switchover swift #1] async writes to codfw (duration: 00m 28s) |
[production] |
| 15:15 |
<godog> |
[switchover swift #2] upload backends from codfw to eqiad |
[production] |
| 15:09 |
<godog> |
[switchover swift #3] upload codfw to eqiad |
[production] |
| 15:05 |
<godog> |
[switchover swift #4] repool upload eqiad in dns |
[production] |
| 15:01 |
<godog> |
[switchover swift #5] upload esams to eqiad |
[production] |
| 14:57 |
<godog> |
[switchover swift #6] upload eqiad to 'direct' |
[production] |
| 14:28 |
<paravoid> |
traffic/mediawiki codfw->eqiad switchover is done |
[production] |
| 14:25 |
<_joe_> |
[switchover #8/#3] Running rebuildEntityPerPage.php |
[production] |
| 14:25 |
<_joe_> |
[switchover #8/#2] Enabling crons in eqiad |
[production] |
| 14:24 |
<ori> |
[switchover #8/#1] Starting the jobqueue in eqiad |
[production] |
| 14:23 |
<volans> |
[switchover #8/#3] Re-enable puppet on all eqiad and codfw databases masters |
[production] |
| 14:21 |
<ori@tin> |
Synchronized wmf-config/db-eqiad.php: [switchover #7/#1] Iac92c8bc6b: Put eqiad in read-write mode for datacenter switchover to eqiad (duration: 00m 39s) |
[production] |
| 14:20 |
<volans> |
[switchover #6/#1] Set database masters RW in eqiad for s1-7, es2-3 and x1 |
[production] |
| 14:19 |
<bblack> |
[switchover #5/#6] Switch Varnish MW backends to eqiad - DONE (confirmed) |
[production] |
| 14:18 |
<godog> |
[switchover #5/#7] finished swift-proxy roll restart |
[production] |
| 14:15 |
<bblack> |
[switchover #5/#6] Switch Varnish MW backends to eqiad - starting |
[production] |
| 14:14 |
<mobrovac> |
[switchover #5/2] restbase {{done}} |
[production] |
| 14:10 |
<godog> |
[switchover #5/#7] roll-restart swift-proxy in eqiad and codfw |
[production] |
| 14:09 |
<akosiaris> |
[switchover #5/#3] Misc services cluster (for the action API endpoint) |
[production] |
| 14:09 |
<mobrovac> |
[switchover #5/2] restbase puppet agent -tv && systemctl restart restbase |
[production] |
| 14:07 |
<volans> |
[switchover #5/#5] Switch parsercache RO/RW |
[production] |
| 14:07 |
<_joe_> |
[switchover #5/1] switching redis replication manually |
[production] |
| 14:07 |
<ori@tin> |
Synchronized wmf-config/CommonSettings.php: [switchover #4/#2] I0e85c3d20: Switch wmfMasterDatacenter to eqiad (duration: 00m 26s) |
[production] |
| 14:06 |
<_joe_> |
[switchover #4/1] puppet merged |
[production] |
| 14:03 |
<volans> |
[switchover #3/#1] Set active site's databases (masters) in read-only mode except parsercache ones. |
[production] |
| 14:01 |
<ori@tin> |
Synchronized wmf-config/db-codfw.php: [switchover #2/#1] Id8b2e7a05: Set codfw databases to read-only mode (duration: 00m 24s) |
[production] |
| 13:55 |
<volans> |
[switchover #1/#6] Switch pt-heartbeat from active site (codfw) to new site (eqiad) masters |
[production] |
| 13:54 |
<ori> |
[switchover #1/2] stopping jobrunners in codfw |
[production] |
| 13:54 |
<_joe_> |
[switchover #1/3] stopping crons on wasat |
[production] |
| 13:52 |
<volans> |
[switchover #1/#5] Set final $master status for databases in advance |
[production] |
| 13:50 |
<volans> |
[switchover #1/#4] Disable puppet on all eqiad and codfw databases masters |
[production] |
| 13:50 |
<paravoid> |
commencing codfw->eqiad datacenter switchover |
[production] |
|
2016-04-20
§
|
| 13:17 |
<jynus> |
[switchover-maintenance] Changing DB slave topology for shard s1 on eqiad T111654 |
[production] |
| 12:59 |
<volans> |
[switchover-maintenance] Changing DB slave topology for shard s4 on eqiad T111654 |
[production] |
| 12:54 |
<volans> |
[switchover-maintenance] Changing DB slave topology for shard s5 on eqiad T111654 |
[production] |
| 12:48 |
<jynus> |
[switchover-maintenance] Changing DB slave topology for shard s3 on eqiad T111654 |
[production] |
| 12:37 |
<volans> |
[switchover-maintenance] Changing DB slave topology for shard s6 on eqiad T111654 |
[production] |
| 12:17 |
<volans> |
[switchover-maintenance] Changing DB slave topology for shard s7 on eqiad T111654 |
[production] |
| 10:42 |
<volans> |
[switchover-maintenance] Restarting db1028 (s7) |
[production] |
| 10:33 |
<jynus> |
[switchover-maintenance] Restarting db1018 |
[production] |
| 10:31 |
<volans> |
[switchover-maintenance] Upgrading TLS for shard s7 on eqiad databases |
[production] |
|
2016-04-19
§
|
| 15:02 |
<_joe_> |
[switchover #13] starting maintenace jobs |
[production] |
| 14:48 |
<ori@tin> |
Synchronized wmf-config/db-codfw.php: [switchover #12] I5e9635b8f4: Set codfw databases in read-write (duration: 00m 35s) |
[production] |
| 14:37 |
<godog> |
[switchover #10] puppet and swift reload finished |
[production] |
| 14:34 |
<jynus> |
[switchover #11] applying $master change for codfw masters |
[production] |
| 14:32 |
<godog> |
[switchover #10] running puppet on ms-fe and reload swift |
[production] |
| 14:25 |
<mobrovac> |
[switchover #7] restbase now uses MW from codfw |
[production] |
| 14:25 |
<bblack> |
[switchover #9] varnish - puppet runs complete - done |
[production] |
| 14:20 |
<bblack> |
[switchover #9] varnish - change merged, puppet runs starting |
[production] |