|
2016-04-19
§
|
| 15:30 |
<bblack> |
[traffic codfw switch #1] - puppet merging text caches -> direct |
[production] |
| 15:25 |
<ori@tin> |
Synchronized wmf-config/ProductionServices.php: Iee2e08df5: Fix codfw redis hostnames [no-op, already synced as live hack] (duration: 00m 36s) |
[production] |
| 15:11 |
<andrewbogott> |
testing the log by logging a test |
[production] |
| 15:10 |
<ori@tin> |
Synchronized wmf-config/ProductionServices.php: live-hack fix for rdb2*.eqiad (duration: 00m 34s) |
[production] |
| 15:02 |
<_joe_> |
[switchover #13] starting maintenace jobs |
[production] |
| 15:00 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: db weight teaking to better process the load (duration: 00m 28s) |
[production] |
| 15:00 |
<jynus> |
applying database weight changes |
[production] |
| 14:48 |
<paravoid> |
sites are read-write again |
[production] |
| 14:48 |
<ori@tin> |
Synchronized wmf-config/db-codfw.php: [switchover #12] I5e9635b8f4: Set codfw databases in read-write (duration: 00m 35s) |
[production] |
| 14:43 |
<jynus> |
[swithchover #11-2] Set and confirmed codfw master dbs in read-write |
[production] |
| 14:37 |
<godog> |
[switchover #10] puppet and swift reload finished |
[production] |
| 14:34 |
<jynus> |
[switchover #11] applying $master change for codfw masters |
[production] |
| 14:32 |
<godog> |
[switchover #10] running puppet on ms-fe and reload swift |
[production] |
| 14:25 |
<mobrovac> |
[switchover #7] restbase now uses MW from codfw |
[production] |
| 14:25 |
<bblack> |
[switchover #9] varnish - puppet runs complete - done |
[production] |
| 14:20 |
<bblack> |
[switchover #9] varnish - change merged, puppet runs starting |
[production] |
| 14:19 |
<_joe_> |
[switchover #7] memcached redises are now masters in codfw, running puppet on eqiad to start replicating |
[production] |
| 14:19 |
<akosiaris> |
[switchover #8] restarting parsoid on all wtp nodes |
[production] |
| 14:19 |
<subbu> |
manually restarted parsoid on wtp1001 and confirmed html identical before/after switchover on enwiki:Hospet |
[production] |
| 14:16 |
<ori@tin> |
Synchronized wmf-config/CommonSettings.php: Idbfb0184d: Switch wmfMasterDatacenter to codfw (duration: 00m 30s) |
[production] |
| 14:16 |
<akosiaris> |
[switchover #7] puppet agent -t -v on SCA, SCB cluster |
[production] |
| 14:15 |
<mobrovac> |
[switchover #7] puppet agent -tv && restbase restart |
[production] |
| 14:15 |
<ori> |
[switchover #7] Switch wmfMasterDatacenter to codfw (https://gerrit.wikimedia.org/r/#/c/282897/) |
[production] |
| 14:14 |
<_joe_> |
[switchover #7] running puppet on mc* hosts in codfw |
[production] |
| 14:14 |
<paravoid> |
[switchover #7] setging mediawiki master datacenter to codfw in puppet |
[production] |
| 14:10 |
<_joe_> |
[switchover #6] wiped memcached |
[production] |
| 14:10 |
<jynus> |
[switchover #5] DB Masters on eqiad set as read-only, and confirmed it |
[production] |
| 14:09 |
<_joe_> |
[switchover #6] disabled puppet on all redis hosts as a safety measure before inverting replication after the puppet change |
[production] |
| 14:03 |
<paravoid> |
sites in planned readonly-mode, cf. http://blog.wikimedia.org/2016/04/18/wikimedia-server-switch/ |
[production] |
| 14:02 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Set mediawiki-eqiad in read-only mode for datacenter switchover to codfw (duration: 00m 35s) |
[production] |
| 14:01 |
<jynus> |
[switchover #4] Set mediawiki-eqiad in read-only mode for datacenter switchover to codfw |
[production] |
| 13:56 |
<_joe_> |
[switchover #3] disabling cronjobs on terbium |
[production] |
| 13:55 |
<ori> |
[switchover #1]: disabling eqiad jobrunners via "salt -C 'G@cluster:jobrunner and G@site:eqiad' cmd.run 'service jobrunner stop; service jobchron stop;'". |
[production] |
| 13:47 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool one db per shard as a backup (duration: 00m 27s) |
[production] |
| 11:33 |
<volans> |
changing binlog_format to STATEMENT for codfw masters for shards s1-s7 T124699 |
[production] |
| 10:07 |
<jynus> |
updated dns entries about mysql masters |
[production] |
| 09:16 |
<godog> |
shutdown restbase100[56] |
[production] |
| 09:11 |
<godog> |
stop cassandra and restbase on restbase1006 |
[production] |
| 09:06 |
<godog> |
stop compactions on restbase1014 |
[production] |
| 03:13 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool pc1006 (duration: 00m 28s) |
[production] |
| 03:12 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Repool pc2006 (duration: 00m 31s) |
[production] |
| 02:54 |
<ori@tin> |
Synchronized php-1.27.0-wmf.21/includes/api/ApiStashEdit.php: Ie9799f5ea: Segment stash edit cache stats by basis for hit/miss (duration: 00m 39s) |
[production] |
| 02:31 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Tue Apr 19 02:31:46 UTC 2016 (duration 9m 46s) |
[production] |
| 02:22 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.21) (duration: 09m 47s) |
[production] |
| 02:06 |
<urandom> |
systemctl mask cassandra-b on restbase2004.codfw.wmnet (it should not be running) |
[production] |
| 01:34 |
<mutante> |
restbase2004 - starting crashed cassandra-b service |
[production] |
| 01:11 |
<mutante> |
restbase2004 - unit cassandra-b is failed |
[production] |
| 00:42 |
<mutante> |
kraz - signing puppet certs, adding salt keys |
[production] |
| 00:29 |
<mutante> |
kraz.codfw.wmnet - initial install, adding to site |
[production] |