151-200 of 10000 results (38ms)
2017-04-18 ยง
14:08 <switchdc> (oblivian@sarin) END TASK - switchdc.stages.t09_restart_parsoid(codfw, eqiad) Successfully completed [production]
14:04 <elukey> executed CONFIG SET appendfsync no on rdb2005:6479 to test if fsync stalls affect replication - T159850 [production]
13:50 <switchdc> (oblivian@sarin) START TASK - switchdc.stages.t09_restart_parsoid(codfw, eqiad) Rolling restart parsoid in eqiad and codfw [production]
13:35 <switchdc> (oblivian@sarin) END TASK - switchdc.stages.t01_stop_maintenance(codfw, eqiad) Failed to execute [production]
13:35 <switchdc> (oblivian@sarin) START TASK - switchdc.stages.t01_stop_maintenance(codfw, eqiad) Stop MediaWiki maintenance in the old master DC [production]
12:32 <moritzm> upgrading labnodepool1001 to Linux 4.9 [production]
12:13 <moritzm> upgraded mw1261 to HHVM 3.18.2+wmf2 [production]
11:39 <switchdc> (volans@sarin) END TASK - switchdc.stages.t09_start_maintenance(codfw, eqiad) Successfully completed [production]
11:38 <switchdc> (volans@sarin) START TASK - switchdc.stages.t09_start_maintenance(codfw, eqiad) Start MediaWiki maintenance in the new master DC [production]
11:37 <switchdc> (volans@sarin) END TASK - switchdc.stages.t09_tendril(codfw, eqiad) Successfully completed [production]
11:37 <switchdc> (volans@sarin) START TASK - switchdc.stages.t09_tendril(codfw, eqiad) Update Tendril configuration for the new masters [production]
11:35 <switchdc> (volans@sarin) END TASK - switchdc.stages.t09_tendril(eqiad, codfw) Successfully completed [production]
11:35 <switchdc> (volans@sarin) START TASK - switchdc.stages.t09_tendril(eqiad, codfw) Update Tendril configuration for the new masters [production]
11:34 <switchdc> (volans@sarin) END TASK - switchdc.stages.t09_tendril(codfw, eqiad) Successfully completed [production]
11:34 <switchdc> (volans@sarin) START TASK - switchdc.stages.t09_tendril(codfw, eqiad) Update Tendril configuration for the new masters [production]
11:33 <switchdc> (volans@sarin) END TASK - switchdc.stages.t09_restore_ttl(codfw, eqiad) Successfully completed [production]
11:33 <switchdc> (volans@sarin) START TASK - switchdc.stages.t09_restore_ttl(codfw, eqiad) Restore the TTL of all the MediaWiki discovery records [production]
11:31 <switchdc> (volans@sarin) END TASK - switchdc.stages.t08_stop_mediawiki_readonly(codfw, eqiad) Successfully completed [production]
11:31 <switchdc> (volans@sarin) START TASK - switchdc.stages.t08_stop_mediawiki_readonly(codfw, eqiad) Set MediaWiki in read-write mode (db_to config already merged and git pulled) [production]
11:30 <switchdc> (volans@sarin) END TASK - switchdc.stages.t07_coredb_masters_readwrite(codfw, eqiad) Successfully completed [production]
11:30 <switchdc> (volans@sarin) START TASK - switchdc.stages.t07_coredb_masters_readwrite(codfw, eqiad) set core DB masters in read-write mode [production]
11:18 <switchdc> (volans@sarin) END TASK - switchdc.stages.t06_redis(codfw, eqiad) Successfully completed [production]
11:18 <switchdc> (volans@sarin) START TASK - switchdc.stages.t06_redis(codfw, eqiad) Switch the Redis replication [production]
11:14 <moritzm> upgrading logstash* to Linux 4.9 [production]
10:58 <switchdc> (volans@sarin) END TASK - switchdc.stages.t05_switch_traffic(codfw, eqiad) Successfully completed [production]
10:56 <switchdc> (volans@sarin) START TASK - switchdc.stages.t05_switch_traffic(codfw, eqiad) Switch traffic flow to the appservers in the new datacenter [production]
10:56 <switchdc> (volans@sarin) END TASK - switchdc.stages.t05_switch_datacenter(codfw, eqiad) Successfully completed [production]
10:55 <switchdc> (volans@sarin) START TASK - switchdc.stages.t05_switch_datacenter(codfw, eqiad) Switch MediaWiki configuration to the new datacenter [production]
10:48 <switchdc> (volans@sarin) END TASK - switchdc.stages.t03_coredb_masters_readonly(codfw, eqiad) Failed to execute [production]
10:48 <switchdc> (volans@sarin) START TASK - switchdc.stages.t03_coredb_masters_readonly(codfw, eqiad) set core DB masters in read-only mode [production]
10:43 <switchdc> (volans@sarin) END TASK - switchdc.stages.t02_start_mediawiki_readonly(codfw, eqiad) Successfully completed [production]
10:43 <switchdc> (volans@sarin) START TASK - switchdc.stages.t02_start_mediawiki_readonly(codfw, eqiad) Set MediaWiki in read-only mode (db_from config already merged and git pulled) [production]
10:33 <switchdc> (volans@sarin) END TASK - switchdc.stages.t01_stop_maintenance(codfw, eqiad) Failed to execute [production]
10:33 <switchdc> (volans@sarin) START TASK - switchdc.stages.t01_stop_maintenance(codfw, eqiad) Stop MediaWiki maintenance in the old master DC [production]
10:31 <switchdc> (volans@sarin) END TASK - switchdc.stages.t00_reduce_ttl(codfw, eqiad) Successfully completed [production]
10:31 <switchdc> (volans@sarin) START TASK - switchdc.stages.t00_reduce_ttl(codfw, eqiad) Reduce the TTL of all the MediaWiki discovery records [production]
10:31 <switchdc> (volans@sarin) END TASK - switchdc.stages.t00_disable_puppet(codfw, eqiad) Successfully completed [production]
10:31 <switchdc> (volans@sarin) START TASK - switchdc.stages.t00_disable_puppet(codfw, eqiad) Disabling puppet on selected hosts [production]
10:28 <switchdc> (volans@sarin) END TASK - switchdc.stages.t00_reduce_ttl(codfw, eqiad) Failed to execute [production]
10:28 <switchdc> (volans@sarin) START TASK - switchdc.stages.t00_reduce_ttl(codfw, eqiad) Reduce the TTL of all the MediaWiki discovery records [production]
10:26 <switchdc> (volans@sarin) END TASK - switchdc.stages.t00_disable_puppet(codfw, eqiad) Successfully completed [production]
10:26 <switchdc> (volans@sarin) START TASK - switchdc.stages.t00_disable_puppet(codfw, eqiad) Disabling puppet on selected hosts [production]
10:25 <volans> Final test of switchdc steps in the codfw->eqiad configuration, only idempotent changes, T160178 [production]
10:25 <moritzm> installing wireshark security updates [production]
10:20 <moritzm> uploaded HHVM 3.18.2+wmf2 for jessie-wikimedia/experimental (includes fix for T162354) [production]
09:52 <oblivian:> Setting zotero in codfw UP [production]
09:50 <_joe_> testing switchover script for services, will act on zotero in codfw [production]
09:45 <_joe_> adding 60G to the ocg output partition on ocg1003 [production]
09:17 <oblivian@neodymium> conftool action : set/pooled=true; selector: dnsdisc=zotero,name=codfw [production]
09:03 <volans> upgrading conftool to v0.4.1 on neodymium/sarin [production]