|
2017-05-02
ยง
|
| 20:04 |
<hashar> |
Restarting Jenkins for plugin rollback |
[production] |
| 17:51 |
<bblack> |
codfw->eqiad switchback: end-user edge traffic back to normal @ eqiad ( https://gerrit.wikimedia.org/r/#/c/351330/ ) - 10 minute TTL for bulk traffic pattern shift starts now. |
[production] |
| 17:50 |
<mobrovac@naos> |
Finished deploy [restbase/deploy@6adb0f2]: Include displaytitle and page_id in the summary output and bump the content type version - T163729 T164079 (duration: 06m 04s) |
[production] |
| 17:48 |
<papaul> |
new db servers signing puppet certs,salt-key, initial run |
[production] |
| 17:44 |
<mobrovac@naos> |
Started deploy [restbase/deploy@6adb0f2]: Include displaytitle and page_id in the summary output and bump the content type version - T163729 T164079 |
[production] |
| 17:40 |
<END> |
(PASS) - Start MediaWiki jobrunners, videoscalers and maintenance in codfw - t09_start_maintenance (switchdc/volans@neodymium) |
[production] |
| 17:39 |
<mobrovac@naos> |
Finished deploy [restbase/deploy@6adb0f2]: (no justification provided) (duration: 01m 34s) |
[production] |
| 17:38 |
<START> |
- Start MediaWiki jobrunners, videoscalers and maintenance in codfw - t09_start_maintenance (switchdc/volans@neodymium) |
[production] |
| 17:37 |
<mobrovac@naos> |
Started deploy [restbase/deploy@6adb0f2]: (no justification provided) |
[production] |
| 17:37 |
<END> |
(PASS) - Restore the TTL of all the MediaWiki read-write discovery records and cleanup confd stale files - t09_restore_ttl (switchdc/volans@neodymium) |
[production] |
| 17:37 |
<START> |
- Restore the TTL of all the MediaWiki read-write discovery records and cleanup confd stale files - t09_restore_ttl (switchdc/volans@neodymium) |
[production] |
| 17:35 |
<END> |
(PASS) - Set MediaWiki in read-write mode in codfw - t08_stop_mediawiki_readonly (switchdc/volans@neodymium) |
[production] |
| 17:35 |
<MediaWiki> |
read-only period ends at: 2017-05-02 17:35:48.111079 (switchdc/volans@neodymium) |
[production] |
| 17:35 |
<START> |
- Set MediaWiki in read-write mode in codfw - t08_stop_mediawiki_readonly (switchdc/volans@neodymium) |
[production] |
| 17:35 |
<oblivian@puppetmaster1001> |
conftool action : set/val=test; selector: name=ReadOnly,scope=codfw |
[production] |
| 17:33 |
<END> |
(PASS) - Set core DB masters in read-write mode in codfw, ensure masters in eqiad are read-only - t07_coredb_masters_readwrite (switchdc/volans@neodymium) |
[production] |
| 17:33 |
<START> |
- Set core DB masters in read-write mode in codfw, ensure masters in eqiad are read-only - t07_coredb_masters_readwrite (switchdc/volans@neodymium) |
[production] |
| 17:32 |
<END> |
(PASS) - Switch the Redis masters from eqiad to codfw and invert the replication - t06_redis (switchdc/volans@neodymium) |
[production] |
| 17:32 |
<START> |
- Switch the Redis masters from eqiad to codfw and invert the replication - t06_redis (switchdc/volans@neodymium) |
[production] |
| 17:31 |
<END> |
(PASS) - Switch MediaWiki master datacenter and read-write discovery records from eqiad to codfw - t05_switch_datacenter (switchdc/volans@neodymium) |
[production] |
| 17:31 |
<START> |
- Switch MediaWiki master datacenter and read-write discovery records from eqiad to codfw - t05_switch_datacenter (switchdc/volans@neodymium) |
[production] |
| 17:23 |
<END> |
(FAIL) - Switch MediaWiki master datacenter and read-write discovery records from eqiad to codfw - t05_switch_datacenter (switchdc/volans@neodymium) |
[production] |
| 17:23 |
<START> |
- Switch MediaWiki master datacenter and read-write discovery records from eqiad to codfw - t05_switch_datacenter (switchdc/volans@neodymium) |
[production] |
| 17:20 |
<END> |
(PASS) - Switch traffic flow to the appservers from eqiad to codfw - t05_switch_traffic (switchdc/volans@neodymium) |
[production] |
| 17:17 |
<START> |
- Switch traffic flow to the appservers from eqiad to codfw - t05_switch_traffic (switchdc/volans@neodymium) |
[production] |
| 17:08 |
<END> |
(FAIL) - Switch MediaWiki master datacenter and read-write discovery records from eqiad to codfw - t05_switch_datacenter (switchdc/volans@neodymium) |
[production] |
| 17:08 |
<START> |
- Switch MediaWiki master datacenter and read-write discovery records from eqiad to codfw - t05_switch_datacenter (switchdc/volans@neodymium) |
[production] |
| 17:05 |
<catrope@naos> |
Synchronized php-1.29.0-wmf.21/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.ArticleTarget.js: T164157 (duration: 01m 00s) |
[production] |
| 17:03 |
<END> |
(FAIL) - Set core DB masters in read-only mode in eqiad, ensure all masters are read-only - t03_coredb_masters_readonly (switchdc/volans@neodymium) |
[production] |
| 17:03 |
<START> |
- Set core DB masters in read-only mode in eqiad, ensure all masters are read-only - t03_coredb_masters_readonly (switchdc/volans@neodymium) |
[production] |
| 16:58 |
<END> |
(FAIL) - Set MediaWiki in read-only mode in eqiad - t02_start_mediawiki_readonly (switchdc/volans@neodymium) |
[production] |
| 16:57 |
<MediaWiki> |
read-only period starts at: 2017-05-02 16:57:37.952132 (switchdc/volans@neodymium) |
[production] |
| 16:57 |
<START> |
- Set MediaWiki in read-only mode in eqiad - t02_start_mediawiki_readonly (switchdc/volans@neodymium) |
[production] |
| 16:56 |
<ppchelko@naos> |
Finished deploy [restbase/deploy@6adb0f2]: Summary endpoint enhancements. Restart after a check timeout (duration: 07m 56s) |
[production] |
| 16:53 |
<END> |
(FAIL) - Stop MediaWiki jobrunners, videoscalers and cronjobs in eqiad - t01_stop_maintenance (switchdc/volans@neodymium) |
[production] |
| 16:53 |
<START> |
- Stop MediaWiki jobrunners, videoscalers and cronjobs in eqiad - t01_stop_maintenance (switchdc/volans@neodymium) |
[production] |
| 16:52 |
<END> |
(PASS) - Disabling puppet on selected hosts in eqiad and codfw - t00_disable_puppet (switchdc/volans@neodymium) |
[production] |
| 16:51 |
<START> |
- Disabling puppet on selected hosts in eqiad and codfw - t00_disable_puppet (switchdc/volans@neodymium) |
[production] |
| 16:51 |
<END> |
(PASS) - Reduce the TTL of all the MediaWiki read-write discovery records - t00_reduce_ttl (switchdc/volans@neodymium) |
[production] |
| 16:50 |
<START> |
- Reduce the TTL of all the MediaWiki read-write discovery records - t00_reduce_ttl (switchdc/volans@neodymium) |
[production] |
| 16:50 |
<END> |
(FAIL) - Reduce the TTL of all the MediaWiki read-write discovery records - t00_reduce_ttl (switchdc/volans@neodymium) |
[production] |
| 16:50 |
<START> |
- Reduce the TTL of all the MediaWiki read-write discovery records - t00_reduce_ttl (switchdc/volans@neodymium) |
[production] |
| 16:48 |
<ppchelko@naos> |
Started deploy [restbase/deploy@6adb0f2]: Summary endpoint enhancements. Restart after a check timeout |
[production] |
| 16:47 |
<volans> |
testing (not dry-run) tasks for tomorrow's switchover in reverse mode eqiad->codfw |
[production] |
| 16:43 |
<ppchelko@naos> |
Started deploy [restbase/deploy@6adb0f2]: Summary endpoint enhancements. Restart after a check fail |
[production] |
| 16:42 |
<ppchelko@naos> |
Finished deploy [restbase/deploy@6adb0f2]: Summary endpoint enhancements (duration: 05m 47s) |
[production] |
| 16:37 |
<ppchelko@naos> |
Started deploy [restbase/deploy@6adb0f2]: Summary endpoint enhancements |
[production] |
| 16:36 |
<END> |
(PASS) - Wipe and warmup caches in codfw - t04_cache_wipe (switchdc/oblivian@neodymium) |
[production] |
| 16:32 |
<END> |
(PASS) - Resync the redis for jobqueues in eqiad with the masters in codfw - t04_resync_redis (switchdc/oblivian@neodymium) |
[production] |
| 16:32 |
<_joe_> |
message about cache warmup is wrong, it is being executed in eqiad |
[production] |