751-800 of 10000 results (37ms)
2017-04-18 ยง
15:02 <gehel> upgrading elastic2020 to elasticsearch 5.1.2 [production]
14:55 <_joe_> switchover of services, misc things done [production]
14:54 <oblivian:> Setting restbase-async in codfw DOWN [production]
14:54 <oblivian:> Setting restbase-async in eqiad UP [production]
14:43 <_joe_> switching traffic for all a/a services plus maps and restbase to codfw-only [production]
14:38 <_joe_> forcing puppet run on caches for catching up with the a/a setting of maps and restbase [production]
14:33 <oblivian:> Setting restbase in eqiad DOWN [production]
14:33 <_joe_> starting switchover of services eqiad => codfw; external traffic will be switched over, as well as internal traffic to restbase [production]
14:25 <gehel> un-ban elastic2020 to get ready for real-life test during switchover - T149006 [production]
14:22 <elukey> executed config set client-output-buffer-limit "normal 0 0 0 slave 2147483648 2147483648 300 pubsub 33554432 8388608 60" on rdb2005:6749 as attempt to solve slave lagging - T159850 [production]
14:21 <oblivian:> Setting mobileapps in eqiad UP [production]
14:14 <oblivian:> Setting mobileapps in eqiad DOWN [production]
14:11 <elukey> executed CONFIG SET appendfsync everysec (default) to restore defaults on rdb2005:6479- T159850 [production]
14:08 <switchdc> (oblivian@sarin) END TASK - switchdc.stages.t09_restart_parsoid(codfw, eqiad) Successfully completed [production]
14:04 <elukey> executed CONFIG SET appendfsync no on rdb2005:6479 to test if fsync stalls affect replication - T159850 [production]
13:50 <switchdc> (oblivian@sarin) START TASK - switchdc.stages.t09_restart_parsoid(codfw, eqiad) Rolling restart parsoid in eqiad and codfw [production]
13:35 <switchdc> (oblivian@sarin) END TASK - switchdc.stages.t01_stop_maintenance(codfw, eqiad) Failed to execute [production]
13:35 <switchdc> (oblivian@sarin) START TASK - switchdc.stages.t01_stop_maintenance(codfw, eqiad) Stop MediaWiki maintenance in the old master DC [production]
12:32 <moritzm> upgrading labnodepool1001 to Linux 4.9 [production]
12:13 <moritzm> upgraded mw1261 to HHVM 3.18.2+wmf2 [production]
11:39 <switchdc> (volans@sarin) END TASK - switchdc.stages.t09_start_maintenance(codfw, eqiad) Successfully completed [production]
11:38 <switchdc> (volans@sarin) START TASK - switchdc.stages.t09_start_maintenance(codfw, eqiad) Start MediaWiki maintenance in the new master DC [production]
11:37 <switchdc> (volans@sarin) END TASK - switchdc.stages.t09_tendril(codfw, eqiad) Successfully completed [production]
11:37 <switchdc> (volans@sarin) START TASK - switchdc.stages.t09_tendril(codfw, eqiad) Update Tendril configuration for the new masters [production]
11:35 <switchdc> (volans@sarin) END TASK - switchdc.stages.t09_tendril(eqiad, codfw) Successfully completed [production]
11:35 <switchdc> (volans@sarin) START TASK - switchdc.stages.t09_tendril(eqiad, codfw) Update Tendril configuration for the new masters [production]
11:34 <switchdc> (volans@sarin) END TASK - switchdc.stages.t09_tendril(codfw, eqiad) Successfully completed [production]
11:34 <switchdc> (volans@sarin) START TASK - switchdc.stages.t09_tendril(codfw, eqiad) Update Tendril configuration for the new masters [production]
11:33 <switchdc> (volans@sarin) END TASK - switchdc.stages.t09_restore_ttl(codfw, eqiad) Successfully completed [production]
11:33 <switchdc> (volans@sarin) START TASK - switchdc.stages.t09_restore_ttl(codfw, eqiad) Restore the TTL of all the MediaWiki discovery records [production]
11:31 <switchdc> (volans@sarin) END TASK - switchdc.stages.t08_stop_mediawiki_readonly(codfw, eqiad) Successfully completed [production]
11:31 <switchdc> (volans@sarin) START TASK - switchdc.stages.t08_stop_mediawiki_readonly(codfw, eqiad) Set MediaWiki in read-write mode (db_to config already merged and git pulled) [production]
11:30 <switchdc> (volans@sarin) END TASK - switchdc.stages.t07_coredb_masters_readwrite(codfw, eqiad) Successfully completed [production]
11:30 <switchdc> (volans@sarin) START TASK - switchdc.stages.t07_coredb_masters_readwrite(codfw, eqiad) set core DB masters in read-write mode [production]
11:18 <switchdc> (volans@sarin) END TASK - switchdc.stages.t06_redis(codfw, eqiad) Successfully completed [production]
11:18 <switchdc> (volans@sarin) START TASK - switchdc.stages.t06_redis(codfw, eqiad) Switch the Redis replication [production]
11:14 <moritzm> upgrading logstash* to Linux 4.9 [production]
10:58 <switchdc> (volans@sarin) END TASK - switchdc.stages.t05_switch_traffic(codfw, eqiad) Successfully completed [production]
10:56 <switchdc> (volans@sarin) START TASK - switchdc.stages.t05_switch_traffic(codfw, eqiad) Switch traffic flow to the appservers in the new datacenter [production]
10:56 <switchdc> (volans@sarin) END TASK - switchdc.stages.t05_switch_datacenter(codfw, eqiad) Successfully completed [production]
10:55 <switchdc> (volans@sarin) START TASK - switchdc.stages.t05_switch_datacenter(codfw, eqiad) Switch MediaWiki configuration to the new datacenter [production]
10:48 <switchdc> (volans@sarin) END TASK - switchdc.stages.t03_coredb_masters_readonly(codfw, eqiad) Failed to execute [production]
10:48 <switchdc> (volans@sarin) START TASK - switchdc.stages.t03_coredb_masters_readonly(codfw, eqiad) set core DB masters in read-only mode [production]
10:43 <switchdc> (volans@sarin) END TASK - switchdc.stages.t02_start_mediawiki_readonly(codfw, eqiad) Successfully completed [production]
10:43 <switchdc> (volans@sarin) START TASK - switchdc.stages.t02_start_mediawiki_readonly(codfw, eqiad) Set MediaWiki in read-only mode (db_from config already merged and git pulled) [production]
10:33 <switchdc> (volans@sarin) END TASK - switchdc.stages.t01_stop_maintenance(codfw, eqiad) Failed to execute [production]
10:33 <switchdc> (volans@sarin) START TASK - switchdc.stages.t01_stop_maintenance(codfw, eqiad) Stop MediaWiki maintenance in the old master DC [production]
10:31 <switchdc> (volans@sarin) END TASK - switchdc.stages.t00_reduce_ttl(codfw, eqiad) Successfully completed [production]
10:31 <switchdc> (volans@sarin) START TASK - switchdc.stages.t00_reduce_ttl(codfw, eqiad) Reduce the TTL of all the MediaWiki discovery records [production]
10:31 <switchdc> (volans@sarin) END TASK - switchdc.stages.t00_disable_puppet(codfw, eqiad) Successfully completed [production]