901-950 of 10000 results (31ms)
2017-05-02 ยง
17:37 <mobrovac@naos> Started deploy [restbase/deploy@6adb0f2]: (no justification provided) [production]
17:37 <END> (PASS) - Restore the TTL of all the MediaWiki read-write discovery records and cleanup confd stale files - t09_restore_ttl (switchdc/volans@neodymium) [production]
17:37 <START> - Restore the TTL of all the MediaWiki read-write discovery records and cleanup confd stale files - t09_restore_ttl (switchdc/volans@neodymium) [production]
17:35 <END> (PASS) - Set MediaWiki in read-write mode in codfw - t08_stop_mediawiki_readonly (switchdc/volans@neodymium) [production]
17:35 <MediaWiki> read-only period ends at: 2017-05-02 17:35:48.111079 (switchdc/volans@neodymium) [production]
17:35 <START> - Set MediaWiki in read-write mode in codfw - t08_stop_mediawiki_readonly (switchdc/volans@neodymium) [production]
17:35 <oblivian@puppetmaster1001> conftool action : set/val=test; selector: name=ReadOnly,scope=codfw [production]
17:33 <END> (PASS) - Set core DB masters in read-write mode in codfw, ensure masters in eqiad are read-only - t07_coredb_masters_readwrite (switchdc/volans@neodymium) [production]
17:33 <START> - Set core DB masters in read-write mode in codfw, ensure masters in eqiad are read-only - t07_coredb_masters_readwrite (switchdc/volans@neodymium) [production]
17:32 <END> (PASS) - Switch the Redis masters from eqiad to codfw and invert the replication - t06_redis (switchdc/volans@neodymium) [production]
17:32 <START> - Switch the Redis masters from eqiad to codfw and invert the replication - t06_redis (switchdc/volans@neodymium) [production]
17:31 <END> (PASS) - Switch MediaWiki master datacenter and read-write discovery records from eqiad to codfw - t05_switch_datacenter (switchdc/volans@neodymium) [production]
17:31 <START> - Switch MediaWiki master datacenter and read-write discovery records from eqiad to codfw - t05_switch_datacenter (switchdc/volans@neodymium) [production]
17:23 <END> (FAIL) - Switch MediaWiki master datacenter and read-write discovery records from eqiad to codfw - t05_switch_datacenter (switchdc/volans@neodymium) [production]
17:23 <START> - Switch MediaWiki master datacenter and read-write discovery records from eqiad to codfw - t05_switch_datacenter (switchdc/volans@neodymium) [production]
17:20 <END> (PASS) - Switch traffic flow to the appservers from eqiad to codfw - t05_switch_traffic (switchdc/volans@neodymium) [production]
17:17 <START> - Switch traffic flow to the appservers from eqiad to codfw - t05_switch_traffic (switchdc/volans@neodymium) [production]
17:08 <END> (FAIL) - Switch MediaWiki master datacenter and read-write discovery records from eqiad to codfw - t05_switch_datacenter (switchdc/volans@neodymium) [production]
17:08 <START> - Switch MediaWiki master datacenter and read-write discovery records from eqiad to codfw - t05_switch_datacenter (switchdc/volans@neodymium) [production]
17:05 <catrope@naos> Synchronized php-1.29.0-wmf.21/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.ArticleTarget.js: T164157 (duration: 01m 00s) [production]
17:03 <END> (FAIL) - Set core DB masters in read-only mode in eqiad, ensure all masters are read-only - t03_coredb_masters_readonly (switchdc/volans@neodymium) [production]
17:03 <START> - Set core DB masters in read-only mode in eqiad, ensure all masters are read-only - t03_coredb_masters_readonly (switchdc/volans@neodymium) [production]
16:58 <END> (FAIL) - Set MediaWiki in read-only mode in eqiad - t02_start_mediawiki_readonly (switchdc/volans@neodymium) [production]
16:57 <MediaWiki> read-only period starts at: 2017-05-02 16:57:37.952132 (switchdc/volans@neodymium) [production]
16:57 <START> - Set MediaWiki in read-only mode in eqiad - t02_start_mediawiki_readonly (switchdc/volans@neodymium) [production]
16:56 <ppchelko@naos> Finished deploy [restbase/deploy@6adb0f2]: Summary endpoint enhancements. Restart after a check timeout (duration: 07m 56s) [production]
16:53 <END> (FAIL) - Stop MediaWiki jobrunners, videoscalers and cronjobs in eqiad - t01_stop_maintenance (switchdc/volans@neodymium) [production]
16:53 <START> - Stop MediaWiki jobrunners, videoscalers and cronjobs in eqiad - t01_stop_maintenance (switchdc/volans@neodymium) [production]
16:52 <END> (PASS) - Disabling puppet on selected hosts in eqiad and codfw - t00_disable_puppet (switchdc/volans@neodymium) [production]
16:51 <START> - Disabling puppet on selected hosts in eqiad and codfw - t00_disable_puppet (switchdc/volans@neodymium) [production]
16:51 <END> (PASS) - Reduce the TTL of all the MediaWiki read-write discovery records - t00_reduce_ttl (switchdc/volans@neodymium) [production]
16:50 <START> - Reduce the TTL of all the MediaWiki read-write discovery records - t00_reduce_ttl (switchdc/volans@neodymium) [production]
16:50 <END> (FAIL) - Reduce the TTL of all the MediaWiki read-write discovery records - t00_reduce_ttl (switchdc/volans@neodymium) [production]
16:50 <START> - Reduce the TTL of all the MediaWiki read-write discovery records - t00_reduce_ttl (switchdc/volans@neodymium) [production]
16:48 <ppchelko@naos> Started deploy [restbase/deploy@6adb0f2]: Summary endpoint enhancements. Restart after a check timeout [production]
16:47 <volans> testing (not dry-run) tasks for tomorrow's switchover in reverse mode eqiad->codfw [production]
16:43 <ppchelko@naos> Started deploy [restbase/deploy@6adb0f2]: Summary endpoint enhancements. Restart after a check fail [production]
16:42 <ppchelko@naos> Finished deploy [restbase/deploy@6adb0f2]: Summary endpoint enhancements (duration: 05m 47s) [production]
16:37 <ppchelko@naos> Started deploy [restbase/deploy@6adb0f2]: Summary endpoint enhancements [production]
16:36 <END> (PASS) - Wipe and warmup caches in codfw - t04_cache_wipe (switchdc/oblivian@neodymium) [production]
16:32 <END> (PASS) - Resync the redis for jobqueues in eqiad with the masters in codfw - t04_resync_redis (switchdc/oblivian@neodymium) [production]
16:32 <_joe_> message about cache warmup is wrong, it is being executed in eqiad [production]
16:29 <START> - Resync the redis for jobqueues in eqiad with the masters in codfw - t04_resync_redis (switchdc/oblivian@neodymium) [production]
16:29 <START> - Wipe and warmup caches in codfw - t04_cache_wipe (switchdc/oblivian@neodymium) [production]
16:29 <_joe_> testing (not dry-run) cache wipe/warmup and redis resync for the switchover codfw->eqiad [production]
16:25 <papaul> OS install on new db servers [production]
16:16 <elukey@naos> Synchronized wmf-config/ProductionServices.php: Replace Redis lock IPs after hw refresh (duration: 01m 16s) [production]
15:53 <oblivian@puppetmaster1001> conftool action : set/@read-only.yaml; selector: name=ReadOnly,scope=eqiad [production]
15:36 <ema> cache_misc: upgrade varnish to 4.1.6-1wm1 [production]
15:24 <_joe_> restarting confd in eqiad/esams to pick up the server change [production]