451-500 of 10000 results (37ms)
2017-05-03 ยง
14:54 <gehel> restart of elasticsearch on relforge [production]
14:43 <END> (PASS) - Rolling restart of parsoid in codfw and eqiad - t09_restart_parsoid (switchdc/oblivian@neodymium) [production]
14:27 <START> - Rolling restart of parsoid in codfw and eqiad - t09_restart_parsoid (switchdc/oblivian@neodymium) [production]
14:26 <END> (PASS) - Update Tendril tree to start from the core DB masters in eqiad - t09_tendril (switchdc/oblivian@neodymium) [production]
14:25 <START> - Update Tendril tree to start from the core DB masters in eqiad - t09_tendril (switchdc/oblivian@neodymium) [production]
14:25 <godog> start swiftrepl on ms-fe1005 [production]
14:24 <END> (PASS) - Start MediaWiki jobrunners, videoscalers and maintenance in eqiad - t09_start_maintenance (switchdc/oblivian@neodymium) [production]
14:22 <START> - Start MediaWiki jobrunners, videoscalers and maintenance in eqiad - t09_start_maintenance (switchdc/oblivian@neodymium) [production]
14:21 <END> (PASS) - Restore the TTL of all the MediaWiki read-write discovery records and cleanup confd stale files - t09_restore_ttl (switchdc/oblivian@neodymium) [production]
14:21 <START> - Restore the TTL of all the MediaWiki read-write discovery records and cleanup confd stale files - t09_restore_ttl (switchdc/oblivian@neodymium) [production]
14:20 <END> (PASS) - Set MediaWiki in read-write mode in eqiad (db-eqiad config already merged and git pulled) - t08_stop_mediawiki_readonly (switchdc/oblivian@neodymium) [production]
14:20 <MediaWiki> read-only period ends at: 2017-05-03 14:20:28.286697 (switchdc/oblivian@neodymium) [production]
14:20 <root@naos> Synchronized wmf-config/db-eqiad.php: Set MediaWiki in read-write mode in datacenter eqiad (duration: 00m 32s) [production]
14:19 <START> - Set MediaWiki in read-write mode in eqiad (db-eqiad config already merged and git pulled) - t08_stop_mediawiki_readonly (switchdc/oblivian@neodymium) [production]
14:19 <END> (PASS) - Set core DB masters in read-write mode in eqiad, ensure masters in codfw are read-only - t07_coredb_masters_readwrite (switchdc/oblivian@neodymium) [production]
14:19 <START> - Set core DB masters in read-write mode in eqiad, ensure masters in codfw are read-only - t07_coredb_masters_readwrite (switchdc/oblivian@neodymium) [production]
14:19 <END> (PASS) - Switch the Redis masters from codfw to eqiad and invert the replication - t06_redis (switchdc/oblivian@neodymium) [production]
14:19 <START> - Switch the Redis masters from codfw to eqiad and invert the replication - t06_redis (switchdc/oblivian@neodymium) [production]
14:18 <END> (PASS) - Switch traffic flow to the appservers from codfw to eqiad - t05_switch_traffic (switchdc/oblivian@neodymium) [production]
14:17 <START> - Switch traffic flow to the appservers from codfw to eqiad - t05_switch_traffic (switchdc/oblivian@neodymium) [production]
14:16 <END> (FAIL) - Switch MediaWiki master datacenter and read-write discovery records from codfw to eqiad - t05_switch_datacenter (switchdc/oblivian@neodymium) [production]
14:16 <root@naos> Synchronized wmf-config/CommonSettings.php: Switch MediaWiki active datacenter to eqiad (duration: 00m 31s) [production]
14:15 <START> - Switch MediaWiki master datacenter and read-write discovery records from codfw to eqiad - t05_switch_datacenter (switchdc/oblivian@neodymium) [production]
14:15 <END> (PASS) - Wipe and warmup caches in eqiad - t04_cache_wipe (switchdc/oblivian@neodymium) [production]
14:12 <elukey> restart kafka-mirror-main-eqiad_to_analytics.service on kafka1012 [production]
14:12 <END> (PASS) - Resync the redis for jobqueues in eqiad with the masters in codfw - t04_resync_redis (switchdc/oblivian@neodymium) [production]
14:09 <START> - Wipe and warmup caches in eqiad - t04_cache_wipe (switchdc/oblivian@neodymium) [production]
14:08 <START> - Resync the redis for jobqueues in eqiad with the masters in codfw - t04_resync_redis (switchdc/oblivian@neodymium) [production]
14:08 <END> (PASS) - Set core DB masters in read-only mode in codfw, ensure all masters are read-only - t03_coredb_masters_readonly (switchdc/oblivian@neodymium) [production]
14:08 <START> - Set core DB masters in read-only mode in codfw, ensure all masters are read-only - t03_coredb_masters_readonly (switchdc/oblivian@neodymium) [production]
14:08 <END> (PASS) - Set MediaWiki in read-only mode in codfw (db-codfw config already merged and git pulled) - t02_start_mediawiki_readonly (switchdc/oblivian@neodymium) [production]
14:07 <root@naos> Synchronized wmf-config/db-codfw.php: Set MediaWiki in read-only mode in datacenter codfw (duration: 00m 45s) [production]
14:07 <MediaWiki> read-only period starts at: 2017-05-03 14:07:08.261300 (switchdc/oblivian@neodymium) [production]
14:07 <START> - Set MediaWiki in read-only mode in codfw (db-codfw config already merged and git pulled) - t02_start_mediawiki_readonly (switchdc/oblivian@neodymium) [production]
14:06 <END> (PASS) - Stop MediaWiki jobrunners, videoscalers and cronjobs in codfw - t01_stop_maintenance (switchdc/oblivian@neodymium) [production]
14:01 <START> - Stop MediaWiki jobrunners, videoscalers and cronjobs in codfw - t01_stop_maintenance (switchdc/oblivian@neodymium) [production]
14:00 <godog> stop swiftrepl on ms-fe1005 [production]
13:59 <END> (PASS) - Reduce the TTL of all the MediaWiki read-write discovery records - t00_reduce_ttl (switchdc/oblivian@neodymium) [production]
13:59 <START> - Reduce the TTL of all the MediaWiki read-write discovery records - t00_reduce_ttl (switchdc/oblivian@neodymium) [production]
13:59 <END> (PASS) - Disabling puppet on selected hosts in codfw and eqiad - t00_disable_puppet (switchdc/oblivian@neodymium) [production]
13:58 <START> - Disabling puppet on selected hosts in codfw and eqiad - t00_disable_puppet (switchdc/oblivian@neodymium) [production]
13:16 <hashar> Restarting Jenkins [production]
13:06 <marostegui> db1028: Increased /srv/ by 20G to clear the warning [production]
11:59 <moritzm> rebooted kubernetes1002, not 1003 [production]
11:59 <moritzm> rebooting kubernetes1003 for update to Linux 4.9 [production]
11:39 <moritzm> rebooting kubernetes1001 for update to Linux 4.9 [production]
11:37 <oblivian@naos> Synchronized wmf-config: Changing the read-only reason for the DC switchover (T164177) (duration: 01m 20s) [production]
11:25 <moritzm> uploaded nodepool 0.1.1+wmf7 to apt.wikimedia.org [production]
11:23 <hashar> Upgrading Jenkins 2.46.1 -> 2.46.2 - T144106 [production]
11:16 <jynus> restarting replication on s*, and x1 eqiad -> codfw [production]