|
2016-04-19
ยง
|
| 17:01 |
<ostriches> |
ytterbium: stopped puppet for a bit, testing host key mess. |
[production] |
| 16:55 |
<ostriches> |
restarting gerrit to pick up furud's rsa key |
[production] |
| 15:49 |
<bblack> |
[traffic codfw switch #4] - puppet change complete - done |
[production] |
| 15:49 |
<bblack> |
[traffic codfw switch #2] - confirmed bulk of traffic moved after ~10min for DNS TTL, rates levelling out on eqiad+codfw front network stats |
[production] |
| 15:46 |
<bblack> |
[traffic codfw switch #4] - salting puppet change |
[production] |
| 15:45 |
<bblack> |
[traffic codfw switch #4] - puppet merging eqiad text -> codfw |
[production] |
| 15:41 |
<bblack> |
[traffic codfw switch #3] - puppet change complete - done |
[production] |
| 15:39 |
<bblack> |
[traffic codfw switch #3] - salting puppet change |
[production] |
| 15:38 |
<bblack> |
[traffic codfw switch #3] - puppet merging esams text -> codfw |
[production] |
| 15:35 |
<bblack> |
[traffic codfw switch #2] - authdns-update complete, user traffic to eqiad frontends should start dropping off now |
[production] |
| 15:34 |
<bblack> |
[traffic codfw switch #1] - puppet change complete - done |
[production] |
| 15:31 |
<bblack> |
[traffic codfw switch #1] - salting puppet change |
[production] |
| 15:30 |
<bblack> |
[traffic codfw switch #1] - puppet merging text caches -> direct |
[production] |
| 15:25 |
<ori@tin> |
Synchronized wmf-config/ProductionServices.php: Iee2e08df5: Fix codfw redis hostnames [no-op, already synced as live hack] (duration: 00m 36s) |
[production] |
| 15:11 |
<andrewbogott> |
testing the log by logging a test |
[production] |
| 15:10 |
<ori@tin> |
Synchronized wmf-config/ProductionServices.php: live-hack fix for rdb2*.eqiad (duration: 00m 34s) |
[production] |
| 15:02 |
<_joe_> |
[switchover #13] starting maintenace jobs |
[production] |
| 15:00 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: db weight teaking to better process the load (duration: 00m 28s) |
[production] |
| 15:00 |
<jynus> |
applying database weight changes |
[production] |
| 14:48 |
<paravoid> |
sites are read-write again |
[production] |
| 14:48 |
<ori@tin> |
Synchronized wmf-config/db-codfw.php: [switchover #12] I5e9635b8f4: Set codfw databases in read-write (duration: 00m 35s) |
[production] |
| 14:43 |
<jynus> |
[swithchover #11-2] Set and confirmed codfw master dbs in read-write |
[production] |
| 14:37 |
<godog> |
[switchover #10] puppet and swift reload finished |
[production] |
| 14:34 |
<jynus> |
[switchover #11] applying $master change for codfw masters |
[production] |
| 14:32 |
<godog> |
[switchover #10] running puppet on ms-fe and reload swift |
[production] |
| 14:25 |
<mobrovac> |
[switchover #7] restbase now uses MW from codfw |
[production] |
| 14:25 |
<bblack> |
[switchover #9] varnish - puppet runs complete - done |
[production] |
| 14:20 |
<bblack> |
[switchover #9] varnish - change merged, puppet runs starting |
[production] |
| 14:19 |
<_joe_> |
[switchover #7] memcached redises are now masters in codfw, running puppet on eqiad to start replicating |
[production] |
| 14:19 |
<akosiaris> |
[switchover #8] restarting parsoid on all wtp nodes |
[production] |
| 14:19 |
<subbu> |
manually restarted parsoid on wtp1001 and confirmed html identical before/after switchover on enwiki:Hospet |
[production] |
| 14:16 |
<ori@tin> |
Synchronized wmf-config/CommonSettings.php: Idbfb0184d: Switch wmfMasterDatacenter to codfw (duration: 00m 30s) |
[production] |
| 14:16 |
<akosiaris> |
[switchover #7] puppet agent -t -v on SCA, SCB cluster |
[production] |
| 14:15 |
<mobrovac> |
[switchover #7] puppet agent -tv && restbase restart |
[production] |
| 14:15 |
<ori> |
[switchover #7] Switch wmfMasterDatacenter to codfw (https://gerrit.wikimedia.org/r/#/c/282897/) |
[production] |
| 14:14 |
<_joe_> |
[switchover #7] running puppet on mc* hosts in codfw |
[production] |
| 14:14 |
<paravoid> |
[switchover #7] setging mediawiki master datacenter to codfw in puppet |
[production] |
| 14:10 |
<_joe_> |
[switchover #6] wiped memcached |
[production] |
| 14:10 |
<jynus> |
[switchover #5] DB Masters on eqiad set as read-only, and confirmed it |
[production] |
| 14:09 |
<_joe_> |
[switchover #6] disabled puppet on all redis hosts as a safety measure before inverting replication after the puppet change |
[production] |
| 14:03 |
<paravoid> |
sites in planned readonly-mode, cf. http://blog.wikimedia.org/2016/04/18/wikimedia-server-switch/ |
[production] |
| 14:02 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Set mediawiki-eqiad in read-only mode for datacenter switchover to codfw (duration: 00m 35s) |
[production] |
| 14:01 |
<jynus> |
[switchover #4] Set mediawiki-eqiad in read-only mode for datacenter switchover to codfw |
[production] |
| 13:56 |
<_joe_> |
[switchover #3] disabling cronjobs on terbium |
[production] |
| 13:55 |
<ori> |
[switchover #1]: disabling eqiad jobrunners via "salt -C 'G@cluster:jobrunner and G@site:eqiad' cmd.run 'service jobrunner stop; service jobchron stop;'". |
[production] |
| 13:47 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool one db per shard as a backup (duration: 00m 27s) |
[production] |
| 11:33 |
<volans> |
changing binlog_format to STATEMENT for codfw masters for shards s1-s7 T124699 |
[production] |
| 10:07 |
<jynus> |
updated dns entries about mysql masters |
[production] |
| 09:16 |
<godog> |
shutdown restbase100[56] |
[production] |
| 09:11 |
<godog> |
stop cassandra and restbase on restbase1006 |
[production] |