|
2016-04-21
§
|
| 14:15 |
<bblack> |
[switchover #5/#6] Switch Varnish MW backends to eqiad - starting |
[production] |
| 14:14 |
<mobrovac> |
[switchover #5/2] restbase {{done}} |
[production] |
| 14:10 |
<godog> |
[switchover #5/#7] roll-restart swift-proxy in eqiad and codfw |
[production] |
| 14:09 |
<akosiaris> |
[switchover #5/#3] Misc services cluster (for the action API endpoint) |
[production] |
| 14:09 |
<mobrovac> |
[switchover #5/2] restbase puppet agent -tv && systemctl restart restbase |
[production] |
| 14:07 |
<volans> |
[switchover #5/#5] Switch parsercache RO/RW |
[production] |
| 14:07 |
<_joe_> |
[switchover #5/1] switching redis replication manually |
[production] |
| 14:07 |
<ori@tin> |
Synchronized wmf-config/CommonSettings.php: [switchover #4/#2] I0e85c3d20: Switch wmfMasterDatacenter to eqiad (duration: 00m 26s) |
[production] |
| 14:06 |
<_joe_> |
[switchover #4/1] puppet merged |
[production] |
| 14:03 |
<volans> |
[switchover #3/#1] Set active site's databases (masters) in read-only mode except parsercache ones. |
[production] |
| 14:01 |
<ori@tin> |
Synchronized wmf-config/db-codfw.php: [switchover #2/#1] Id8b2e7a05: Set codfw databases to read-only mode (duration: 00m 24s) |
[production] |
| 13:55 |
<volans> |
[switchover #1/#6] Switch pt-heartbeat from active site (codfw) to new site (eqiad) masters |
[production] |
| 13:54 |
<ori> |
[switchover #1/2] stopping jobrunners in codfw |
[production] |
| 13:54 |
<_joe_> |
[switchover #1/3] stopping crons on wasat |
[production] |
| 13:52 |
<volans> |
[switchover #1/#5] Set final $master status for databases in advance |
[production] |
| 13:50 |
<volans> |
[switchover #1/#4] Disable puppet on all eqiad and codfw databases masters |
[production] |
| 13:50 |
<paravoid> |
commencing codfw->eqiad datacenter switchover |
[production] |
|
2016-04-20
§
|
| 13:17 |
<jynus> |
[switchover-maintenance] Changing DB slave topology for shard s1 on eqiad T111654 |
[production] |
| 12:59 |
<volans> |
[switchover-maintenance] Changing DB slave topology for shard s4 on eqiad T111654 |
[production] |
| 12:54 |
<volans> |
[switchover-maintenance] Changing DB slave topology for shard s5 on eqiad T111654 |
[production] |
| 12:48 |
<jynus> |
[switchover-maintenance] Changing DB slave topology for shard s3 on eqiad T111654 |
[production] |
| 12:37 |
<volans> |
[switchover-maintenance] Changing DB slave topology for shard s6 on eqiad T111654 |
[production] |
| 12:17 |
<volans> |
[switchover-maintenance] Changing DB slave topology for shard s7 on eqiad T111654 |
[production] |
| 10:42 |
<volans> |
[switchover-maintenance] Restarting db1028 (s7) |
[production] |
| 10:33 |
<jynus> |
[switchover-maintenance] Restarting db1018 |
[production] |
| 10:31 |
<volans> |
[switchover-maintenance] Upgrading TLS for shard s7 on eqiad databases |
[production] |
|
2016-04-19
§
|
| 15:02 |
<_joe_> |
[switchover #13] starting maintenace jobs |
[production] |
| 14:48 |
<ori@tin> |
Synchronized wmf-config/db-codfw.php: [switchover #12] I5e9635b8f4: Set codfw databases in read-write (duration: 00m 35s) |
[production] |
| 14:37 |
<godog> |
[switchover #10] puppet and swift reload finished |
[production] |
| 14:34 |
<jynus> |
[switchover #11] applying $master change for codfw masters |
[production] |
| 14:32 |
<godog> |
[switchover #10] running puppet on ms-fe and reload swift |
[production] |
| 14:25 |
<mobrovac> |
[switchover #7] restbase now uses MW from codfw |
[production] |
| 14:25 |
<bblack> |
[switchover #9] varnish - puppet runs complete - done |
[production] |
| 14:20 |
<bblack> |
[switchover #9] varnish - change merged, puppet runs starting |
[production] |
| 14:19 |
<_joe_> |
[switchover #7] memcached redises are now masters in codfw, running puppet on eqiad to start replicating |
[production] |
| 14:19 |
<akosiaris> |
[switchover #8] restarting parsoid on all wtp nodes |
[production] |
| 14:19 |
<subbu> |
manually restarted parsoid on wtp1001 and confirmed html identical before/after switchover on enwiki:Hospet |
[production] |
| 14:16 |
<akosiaris> |
[switchover #7] puppet agent -t -v on SCA, SCB cluster |
[production] |
| 14:15 |
<mobrovac> |
[switchover #7] puppet agent -tv && restbase restart |
[production] |
| 14:15 |
<ori> |
[switchover #7] Switch wmfMasterDatacenter to codfw (https://gerrit.wikimedia.org/r/#/c/282897/) |
[production] |
| 14:14 |
<_joe_> |
[switchover #7] running puppet on mc* hosts in codfw |
[production] |
| 14:14 |
<paravoid> |
[switchover #7] setging mediawiki master datacenter to codfw in puppet |
[production] |
| 14:10 |
<_joe_> |
[switchover #6] wiped memcached |
[production] |
| 14:10 |
<jynus> |
[switchover #5] DB Masters on eqiad set as read-only, and confirmed it |
[production] |
| 14:09 |
<_joe_> |
[switchover #6] disabled puppet on all redis hosts as a safety measure before inverting replication after the puppet change |
[production] |
| 14:02 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Set mediawiki-eqiad in read-only mode for datacenter switchover to codfw (duration: 00m 35s) |
[production] |
| 14:01 |
<jynus> |
[switchover #4] Set mediawiki-eqiad in read-only mode for datacenter switchover to codfw |
[production] |
| 13:56 |
<_joe_> |
[switchover #3] disabling cronjobs on terbium |
[production] |
| 13:55 |
<ori> |
[switchover #1]: disabling eqiad jobrunners via "salt -C 'G@cluster:jobrunner and G@site:eqiad' cmd.run 'service jobrunner stop; service jobchron stop;'". |
[production] |