|
2017-04-19
ยง
|
| 14:46 |
<gehel> |
banning elastic2020 from codfw cluster - T149006 |
[production] |
| 14:46 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t09_restart_parsoid(eqiad, codfw) Rolling restart parsoid in eqiad and codfw |
[production] |
| 14:44 |
<oblivian@tin> |
Synchronized wmf-config/ProductionServices.php: Fix redis locks (duration: 02m 24s) |
[production] |
| 14:41 |
<akosiaris> |
powercycle mw2256 |
[production] |
| 14:33 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t09_tendril(eqiad, codfw) Successfully completed |
[production] |
| 14:33 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t09_tendril(eqiad, codfw) Update Tendril configuration for the new masters |
[production] |
| 14:33 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t09_start_maintenance(eqiad, codfw) Successfully completed |
[production] |
| 14:31 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t09_start_maintenance(eqiad, codfw) Start MediaWiki maintenance in the new master DC |
[production] |
| 14:31 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t09_restore_ttl(eqiad, codfw) Successfully completed |
[production] |
| 14:31 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t09_restore_ttl(eqiad, codfw) Restore the TTL of all the MediaWiki discovery records |
[production] |
| 14:30 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t08_stop_mediawiki_readonly(eqiad, codfw) Successfully completed |
[production] |
| 14:30 |
<switchdc> |
(volans@sarin) MediaWiki read-only period ends at: 2017-04-19 14:30:05.678665 |
[production] |
| 14:30 |
<root@tin> |
Synchronized wmf-config/db-codfw.php: Set MediaWiki in read-write mode in datacenter codfw (duration: 00m 18s) |
[production] |
| 14:29 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t08_stop_mediawiki_readonly(eqiad, codfw) Set MediaWiki in read-write mode (db_to config already merged and git pulled) |
[production] |
| 14:28 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t07_coredb_masters_readwrite(eqiad, codfw) Successfully completed |
[production] |
| 14:28 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t07_coredb_masters_readwrite(eqiad, codfw) set core DB masters in read-write mode |
[production] |
| 14:25 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t06_redis(eqiad, codfw) Successfully completed |
[production] |
| 14:25 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t06_redis(eqiad, codfw) Switch the Redis replication |
[production] |
| 14:25 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t05_switch_traffic(eqiad, codfw) Successfully completed |
[production] |
| 14:22 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t05_switch_traffic(eqiad, codfw) Switch traffic flow to the appservers in the new datacenter |
[production] |
| 14:22 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t05_switch_datacenter(eqiad, codfw) Successfully completed |
[production] |
| 14:22 |
<root@tin> |
Synchronized wmf-config/CommonSettings.php: Switch MediaWiki active datacenter to codfw (duration: 00m 19s) |
[production] |
| 14:21 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t05_switch_datacenter(eqiad, codfw) Switch MediaWiki configuration to the new datacenter |
[production] |
| 14:21 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t04_cache_wipe(eqiad, codfw) Successfully completed |
[production] |
| 14:15 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t04_cache_wipe(eqiad, codfw) wipe and warmup caches |
[production] |
| 14:15 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t03_coredb_masters_readonly(eqiad, codfw) Successfully completed |
[production] |
| 14:15 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t03_coredb_masters_readonly(eqiad, codfw) set core DB masters in read-only mode |
[production] |
| 14:14 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t02_start_mediawiki_readonly(eqiad, codfw) Successfully completed |
[production] |
| 14:14 |
<root@tin> |
Synchronized wmf-config/db-eqiad.php: Set MediaWiki in read-only mode in datacenter eqiad (duration: 01m 29s) |
[production] |
| 14:13 |
<switchdc> |
(volans@sarin) MediaWiki read-only period starts at: 2017-04-19 14:12:54.007017 |
[production] |
| 14:12 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t02_start_mediawiki_readonly(eqiad, codfw) Set MediaWiki in read-only mode (db_from config already merged and git pulled) |
[production] |
| 14:09 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t01_stop_maintenance(eqiad, codfw) Successfully completed |
[production] |
| 14:07 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t01_stop_maintenance(eqiad, codfw) Stop MediaWiki maintenance in the old master DC |
[production] |
| 14:06 |
<godog> |
stop swiftrepl on ms-fe1005 for codfw switchover |
[production] |
| 14:06 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t00_reduce_ttl(eqiad, codfw) Successfully completed |
[production] |
| 14:06 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t00_reduce_ttl(eqiad, codfw) Reduce the TTL of all the MediaWiki discovery records |
[production] |
| 14:06 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t00_disable_puppet(eqiad, codfw) Successfully completed |
[production] |
| 14:05 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t00_disable_puppet(eqiad, codfw) Disabling puppet on selected hosts |
[production] |
| 14:00 |
<bblack@neodymium> |
conftool action : set/pooled=yes; selector: name=cp2014.codfw.wmnet,service=varnish-be |
[production] |
| 13:42 |
<bblack@neodymium> |
conftool action : set/pooled=no; selector: name=cp2014.codfw.wmnet,service=varnish-be |
[production] |
| 13:28 |
<urandom> |
cqlsh -f /etc/cassandra/adduser.cql, recreating user/perms (as-needed) |
[production] |
| 12:38 |
<urandom> |
T163292: Starting removal of Cassandra instance restbase1018-c.eqiad.wmnet |
[production] |
| 11:36 |
<oblivian:> |
Setting swift-rw in eqiad DOWN |
[production] |
| 11:36 |
<oblivian:> |
Setting swift-rw in codfw UP |
[production] |
| 11:36 |
<ema> |
repool varnish-be on cp3044 |
[production] |
| 11:23 |
<godog> |
add naos to git-deploy term on common-infrastructure4 - T162900 |
[production] |
| 11:03 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t04_cache_wipe(eqiad, codfw) Successfully completed |
[production] |
| 10:57 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t04_cache_wipe(eqiad, codfw) wipe and warmup caches |
[production] |
| 10:56 |
<_joe_> |
running the warmup stage in codfw for final testing |
[production] |
| 10:41 |
<ema> |
depool varnish-be on cp3044 because of mailbox lag issues |
[production] |