|
2016-04-19
ยง
|
| 22:42 |
<ori> |
killing rc insert query on db1065 and db1066 |
[production] |
| 21:27 |
<ori> |
running rebuildrecentchanges.php --from=20160419144741 --to=20160419151018 on all wikis |
[production] |
| 21:12 |
<paravoid> |
clearing the exim4 retry database on mx2001 |
[production] |
| 20:44 |
<ori> |
on all wikis, deleting from recentchanges where rc_timestamp > 20160419144741 and rc_timestamp < 20160419151018 |
[production] |
| 20:09 |
<ori> |
ran `mwscript rebuildrecentchanges.php --wiki=testwiki --from=20160419144741 --to=20160419151018` |
[production] |
| 19:53 |
<paravoid> |
staggered varnish bans for 'obj.http.server ~ "^mw2.+"' as a workaround for T133069 |
[production] |
| 19:51 |
<ori@tin> |
Synchronized php-1.27.0-wmf.21/maintenance/rebuildrecentchanges.php: Ie9799f5ea: rebuildrecentchanges: Allow rebuilding specified time range only (duration: 00m 28s) |
[production] |
| 19:43 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Revert "Depool one db server from each shard as a backup" (duration: 00m 27s) |
[production] |
| 19:12 |
<AaronSchulz> |
Cleared enwiki 'enqueue' queue (T133089) |
[production] |
| 19:06 |
<legoktm> |
purging sidebar cache across all wikis (T133069) |
[production] |
| 18:27 |
<mutante> |
kraz.codfw, reinstalling as kraz.wikimedia |
[production] |
| 18:00 |
<_joe_> |
running rebuildEntityPerPage.php on wikidata, T133048 |
[production] |
| 17:49 |
<jynus> |
setting binlog_format=ROW on old x1-master at eqiad (db1029) to reenable replication |
[production] |
| 17:25 |
<demon@tin> |
Synchronized php-1.27.0-wmf.21/extensions/CentralAuth: forgot something (duration: 00m 42s) |
[production] |
| 17:17 |
<volans> |
Deleting pc1003* and pc1006* binlog from pc2006 to make some space |
[production] |
| 17:12 |
<volans> |
Deleting pc1005* binlog from pc2005 to make some space |
[production] |
| 17:08 |
<volans> |
Deleting pc1002* old binlog from pc2005 to make some space |
[production] |
| 17:01 |
<ostriches> |
ytterbium: stopped puppet for a bit, testing host key mess. |
[production] |
| 16:55 |
<ostriches> |
restarting gerrit to pick up furud's rsa key |
[production] |
| 15:49 |
<bblack> |
[traffic codfw switch #4] - puppet change complete - done |
[production] |
| 15:49 |
<bblack> |
[traffic codfw switch #2] - confirmed bulk of traffic moved after ~10min for DNS TTL, rates levelling out on eqiad+codfw front network stats |
[production] |
| 15:46 |
<bblack> |
[traffic codfw switch #4] - salting puppet change |
[production] |
| 15:45 |
<bblack> |
[traffic codfw switch #4] - puppet merging eqiad text -> codfw |
[production] |
| 15:41 |
<bblack> |
[traffic codfw switch #3] - puppet change complete - done |
[production] |
| 15:39 |
<bblack> |
[traffic codfw switch #3] - salting puppet change |
[production] |
| 15:38 |
<bblack> |
[traffic codfw switch #3] - puppet merging esams text -> codfw |
[production] |
| 15:35 |
<bblack> |
[traffic codfw switch #2] - authdns-update complete, user traffic to eqiad frontends should start dropping off now |
[production] |
| 15:34 |
<bblack> |
[traffic codfw switch #1] - puppet change complete - done |
[production] |
| 15:31 |
<bblack> |
[traffic codfw switch #1] - salting puppet change |
[production] |
| 15:30 |
<bblack> |
[traffic codfw switch #1] - puppet merging text caches -> direct |
[production] |
| 15:25 |
<ori@tin> |
Synchronized wmf-config/ProductionServices.php: Iee2e08df5: Fix codfw redis hostnames [no-op, already synced as live hack] (duration: 00m 36s) |
[production] |
| 15:11 |
<andrewbogott> |
testing the log by logging a test |
[production] |
| 15:10 |
<ori@tin> |
Synchronized wmf-config/ProductionServices.php: live-hack fix for rdb2*.eqiad (duration: 00m 34s) |
[production] |
| 15:02 |
<_joe_> |
[switchover #13] starting maintenace jobs |
[production] |
| 15:00 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: db weight teaking to better process the load (duration: 00m 28s) |
[production] |
| 15:00 |
<jynus> |
applying database weight changes |
[production] |
| 14:48 |
<paravoid> |
sites are read-write again |
[production] |
| 14:48 |
<ori@tin> |
Synchronized wmf-config/db-codfw.php: [switchover #12] I5e9635b8f4: Set codfw databases in read-write (duration: 00m 35s) |
[production] |
| 14:43 |
<jynus> |
[swithchover #11-2] Set and confirmed codfw master dbs in read-write |
[production] |
| 14:37 |
<godog> |
[switchover #10] puppet and swift reload finished |
[production] |
| 14:34 |
<jynus> |
[switchover #11] applying $master change for codfw masters |
[production] |
| 14:32 |
<godog> |
[switchover #10] running puppet on ms-fe and reload swift |
[production] |
| 14:25 |
<mobrovac> |
[switchover #7] restbase now uses MW from codfw |
[production] |
| 14:25 |
<bblack> |
[switchover #9] varnish - puppet runs complete - done |
[production] |
| 14:20 |
<bblack> |
[switchover #9] varnish - change merged, puppet runs starting |
[production] |
| 14:19 |
<_joe_> |
[switchover #7] memcached redises are now masters in codfw, running puppet on eqiad to start replicating |
[production] |
| 14:19 |
<akosiaris> |
[switchover #8] restarting parsoid on all wtp nodes |
[production] |
| 14:19 |
<subbu> |
manually restarted parsoid on wtp1001 and confirmed html identical before/after switchover on enwiki:Hospet |
[production] |
| 14:16 |
<ori@tin> |
Synchronized wmf-config/CommonSettings.php: Idbfb0184d: Switch wmfMasterDatacenter to codfw (duration: 00m 30s) |
[production] |
| 14:16 |
<akosiaris> |
[switchover #7] puppet agent -t -v on SCA, SCB cluster |
[production] |