|
2017-04-18
ยง
|
| 14:33 |
<_joe_> |
starting switchover of services eqiad => codfw; external traffic will be switched over, as well as internal traffic to restbase |
[production] |
| 14:25 |
<gehel> |
un-ban elastic2020 to get ready for real-life test during switchover - T149006 |
[production] |
| 14:22 |
<elukey> |
executed config set client-output-buffer-limit "normal 0 0 0 slave 2147483648 2147483648 300 pubsub 33554432 8388608 60" on rdb2005:6749 as attempt to solve slave lagging - T159850 |
[production] |
| 14:21 |
<oblivian:> |
Setting mobileapps in eqiad UP |
[production] |
| 14:14 |
<oblivian:> |
Setting mobileapps in eqiad DOWN |
[production] |
| 14:11 |
<elukey> |
executed CONFIG SET appendfsync everysec (default) to restore defaults on rdb2005:6479- T159850 |
[production] |
| 14:08 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t09_restart_parsoid(codfw, eqiad) Successfully completed |
[production] |
| 14:04 |
<elukey> |
executed CONFIG SET appendfsync no on rdb2005:6479 to test if fsync stalls affect replication - T159850 |
[production] |
| 13:50 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t09_restart_parsoid(codfw, eqiad) Rolling restart parsoid in eqiad and codfw |
[production] |
| 13:35 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t01_stop_maintenance(codfw, eqiad) Failed to execute |
[production] |
| 13:35 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t01_stop_maintenance(codfw, eqiad) Stop MediaWiki maintenance in the old master DC |
[production] |
| 12:32 |
<moritzm> |
upgrading labnodepool1001 to Linux 4.9 |
[production] |
| 12:13 |
<moritzm> |
upgraded mw1261 to HHVM 3.18.2+wmf2 |
[production] |
| 11:39 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t09_start_maintenance(codfw, eqiad) Successfully completed |
[production] |
| 11:38 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t09_start_maintenance(codfw, eqiad) Start MediaWiki maintenance in the new master DC |
[production] |
| 11:37 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t09_tendril(codfw, eqiad) Successfully completed |
[production] |
| 11:37 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t09_tendril(codfw, eqiad) Update Tendril configuration for the new masters |
[production] |
| 11:35 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t09_tendril(eqiad, codfw) Successfully completed |
[production] |
| 11:35 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t09_tendril(eqiad, codfw) Update Tendril configuration for the new masters |
[production] |
| 11:34 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t09_tendril(codfw, eqiad) Successfully completed |
[production] |
| 11:34 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t09_tendril(codfw, eqiad) Update Tendril configuration for the new masters |
[production] |
| 11:33 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t09_restore_ttl(codfw, eqiad) Successfully completed |
[production] |
| 11:33 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t09_restore_ttl(codfw, eqiad) Restore the TTL of all the MediaWiki discovery records |
[production] |
| 11:31 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t08_stop_mediawiki_readonly(codfw, eqiad) Successfully completed |
[production] |
| 11:31 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t08_stop_mediawiki_readonly(codfw, eqiad) Set MediaWiki in read-write mode (db_to config already merged and git pulled) |
[production] |
| 11:30 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t07_coredb_masters_readwrite(codfw, eqiad) Successfully completed |
[production] |
| 11:30 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t07_coredb_masters_readwrite(codfw, eqiad) set core DB masters in read-write mode |
[production] |
| 11:18 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t06_redis(codfw, eqiad) Successfully completed |
[production] |
| 11:18 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t06_redis(codfw, eqiad) Switch the Redis replication |
[production] |
| 11:14 |
<moritzm> |
upgrading logstash* to Linux 4.9 |
[production] |
| 10:58 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t05_switch_traffic(codfw, eqiad) Successfully completed |
[production] |
| 10:56 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t05_switch_traffic(codfw, eqiad) Switch traffic flow to the appservers in the new datacenter |
[production] |
| 10:56 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t05_switch_datacenter(codfw, eqiad) Successfully completed |
[production] |
| 10:55 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t05_switch_datacenter(codfw, eqiad) Switch MediaWiki configuration to the new datacenter |
[production] |
| 10:48 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t03_coredb_masters_readonly(codfw, eqiad) Failed to execute |
[production] |
| 10:48 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t03_coredb_masters_readonly(codfw, eqiad) set core DB masters in read-only mode |
[production] |
| 10:43 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t02_start_mediawiki_readonly(codfw, eqiad) Successfully completed |
[production] |
| 10:43 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t02_start_mediawiki_readonly(codfw, eqiad) Set MediaWiki in read-only mode (db_from config already merged and git pulled) |
[production] |
| 10:33 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t01_stop_maintenance(codfw, eqiad) Failed to execute |
[production] |
| 10:33 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t01_stop_maintenance(codfw, eqiad) Stop MediaWiki maintenance in the old master DC |
[production] |
| 10:31 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t00_reduce_ttl(codfw, eqiad) Successfully completed |
[production] |
| 10:31 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t00_reduce_ttl(codfw, eqiad) Reduce the TTL of all the MediaWiki discovery records |
[production] |
| 10:31 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t00_disable_puppet(codfw, eqiad) Successfully completed |
[production] |
| 10:31 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t00_disable_puppet(codfw, eqiad) Disabling puppet on selected hosts |
[production] |
| 10:28 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t00_reduce_ttl(codfw, eqiad) Failed to execute |
[production] |
| 10:28 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t00_reduce_ttl(codfw, eqiad) Reduce the TTL of all the MediaWiki discovery records |
[production] |
| 10:26 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t00_disable_puppet(codfw, eqiad) Successfully completed |
[production] |
| 10:26 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t00_disable_puppet(codfw, eqiad) Disabling puppet on selected hosts |
[production] |
| 10:25 |
<volans> |
Final test of switchdc steps in the codfw->eqiad configuration, only idempotent changes, T160178 |
[production] |
| 10:25 |
<moritzm> |
installing wireshark security updates |
[production] |