|
2017-05-03
ยง
|
| 16:08 |
<mutante> |
install2002 - temp stop puppet to debug dhcp issue of db2084 |
[production] |
| 15:13 |
<catrope@naos> |
Synchronized php-1.29.0-wmf.21/includes/logging/LogPager.php: Replace FORCE INDEX(ls_field_val) with IGNORE INDEX(ls_log_id) (https://gerrit.wikimedia.org/r/#/c/351653/ for T17441) (duration: 01m 14s) |
[production] |
| 15:09 |
<RoanKattouw> |
Live-hacked (cherry-picked) https://gerrit.wikimedia.org/r/#/c/351653/ onto naos and synced to mwdebug1002 for testing |
[production] |
| 14:54 |
<gehel> |
restart of elasticsearch on relforge |
[production] |
| 14:43 |
<END> |
(PASS) - Rolling restart of parsoid in codfw and eqiad - t09_restart_parsoid (switchdc/oblivian@neodymium) |
[production] |
| 14:27 |
<START> |
- Rolling restart of parsoid in codfw and eqiad - t09_restart_parsoid (switchdc/oblivian@neodymium) |
[production] |
| 14:26 |
<END> |
(PASS) - Update Tendril tree to start from the core DB masters in eqiad - t09_tendril (switchdc/oblivian@neodymium) |
[production] |
| 14:25 |
<START> |
- Update Tendril tree to start from the core DB masters in eqiad - t09_tendril (switchdc/oblivian@neodymium) |
[production] |
| 14:25 |
<godog> |
start swiftrepl on ms-fe1005 |
[production] |
| 14:24 |
<END> |
(PASS) - Start MediaWiki jobrunners, videoscalers and maintenance in eqiad - t09_start_maintenance (switchdc/oblivian@neodymium) |
[production] |
| 14:22 |
<START> |
- Start MediaWiki jobrunners, videoscalers and maintenance in eqiad - t09_start_maintenance (switchdc/oblivian@neodymium) |
[production] |
| 14:21 |
<END> |
(PASS) - Restore the TTL of all the MediaWiki read-write discovery records and cleanup confd stale files - t09_restore_ttl (switchdc/oblivian@neodymium) |
[production] |
| 14:21 |
<START> |
- Restore the TTL of all the MediaWiki read-write discovery records and cleanup confd stale files - t09_restore_ttl (switchdc/oblivian@neodymium) |
[production] |
| 14:20 |
<END> |
(PASS) - Set MediaWiki in read-write mode in eqiad (db-eqiad config already merged and git pulled) - t08_stop_mediawiki_readonly (switchdc/oblivian@neodymium) |
[production] |
| 14:20 |
<MediaWiki> |
read-only period ends at: 2017-05-03 14:20:28.286697 (switchdc/oblivian@neodymium) |
[production] |
| 14:20 |
<root@naos> |
Synchronized wmf-config/db-eqiad.php: Set MediaWiki in read-write mode in datacenter eqiad (duration: 00m 32s) |
[production] |
| 14:19 |
<START> |
- Set MediaWiki in read-write mode in eqiad (db-eqiad config already merged and git pulled) - t08_stop_mediawiki_readonly (switchdc/oblivian@neodymium) |
[production] |
| 14:19 |
<END> |
(PASS) - Set core DB masters in read-write mode in eqiad, ensure masters in codfw are read-only - t07_coredb_masters_readwrite (switchdc/oblivian@neodymium) |
[production] |
| 14:19 |
<START> |
- Set core DB masters in read-write mode in eqiad, ensure masters in codfw are read-only - t07_coredb_masters_readwrite (switchdc/oblivian@neodymium) |
[production] |
| 14:19 |
<END> |
(PASS) - Switch the Redis masters from codfw to eqiad and invert the replication - t06_redis (switchdc/oblivian@neodymium) |
[production] |
| 14:19 |
<START> |
- Switch the Redis masters from codfw to eqiad and invert the replication - t06_redis (switchdc/oblivian@neodymium) |
[production] |
| 14:18 |
<END> |
(PASS) - Switch traffic flow to the appservers from codfw to eqiad - t05_switch_traffic (switchdc/oblivian@neodymium) |
[production] |
| 14:17 |
<START> |
- Switch traffic flow to the appservers from codfw to eqiad - t05_switch_traffic (switchdc/oblivian@neodymium) |
[production] |
| 14:16 |
<END> |
(FAIL) - Switch MediaWiki master datacenter and read-write discovery records from codfw to eqiad - t05_switch_datacenter (switchdc/oblivian@neodymium) |
[production] |
| 14:16 |
<root@naos> |
Synchronized wmf-config/CommonSettings.php: Switch MediaWiki active datacenter to eqiad (duration: 00m 31s) |
[production] |
| 14:15 |
<START> |
- Switch MediaWiki master datacenter and read-write discovery records from codfw to eqiad - t05_switch_datacenter (switchdc/oblivian@neodymium) |
[production] |
| 14:15 |
<END> |
(PASS) - Wipe and warmup caches in eqiad - t04_cache_wipe (switchdc/oblivian@neodymium) |
[production] |
| 14:12 |
<elukey> |
restart kafka-mirror-main-eqiad_to_analytics.service on kafka1012 |
[production] |
| 14:12 |
<END> |
(PASS) - Resync the redis for jobqueues in eqiad with the masters in codfw - t04_resync_redis (switchdc/oblivian@neodymium) |
[production] |
| 14:09 |
<START> |
- Wipe and warmup caches in eqiad - t04_cache_wipe (switchdc/oblivian@neodymium) |
[production] |
| 14:08 |
<START> |
- Resync the redis for jobqueues in eqiad with the masters in codfw - t04_resync_redis (switchdc/oblivian@neodymium) |
[production] |
| 14:08 |
<END> |
(PASS) - Set core DB masters in read-only mode in codfw, ensure all masters are read-only - t03_coredb_masters_readonly (switchdc/oblivian@neodymium) |
[production] |
| 14:08 |
<START> |
- Set core DB masters in read-only mode in codfw, ensure all masters are read-only - t03_coredb_masters_readonly (switchdc/oblivian@neodymium) |
[production] |
| 14:08 |
<END> |
(PASS) - Set MediaWiki in read-only mode in codfw (db-codfw config already merged and git pulled) - t02_start_mediawiki_readonly (switchdc/oblivian@neodymium) |
[production] |
| 14:07 |
<root@naos> |
Synchronized wmf-config/db-codfw.php: Set MediaWiki in read-only mode in datacenter codfw (duration: 00m 45s) |
[production] |
| 14:07 |
<MediaWiki> |
read-only period starts at: 2017-05-03 14:07:08.261300 (switchdc/oblivian@neodymium) |
[production] |
| 14:07 |
<START> |
- Set MediaWiki in read-only mode in codfw (db-codfw config already merged and git pulled) - t02_start_mediawiki_readonly (switchdc/oblivian@neodymium) |
[production] |
| 14:06 |
<END> |
(PASS) - Stop MediaWiki jobrunners, videoscalers and cronjobs in codfw - t01_stop_maintenance (switchdc/oblivian@neodymium) |
[production] |
| 14:01 |
<START> |
- Stop MediaWiki jobrunners, videoscalers and cronjobs in codfw - t01_stop_maintenance (switchdc/oblivian@neodymium) |
[production] |
| 14:00 |
<godog> |
stop swiftrepl on ms-fe1005 |
[production] |
| 13:59 |
<END> |
(PASS) - Reduce the TTL of all the MediaWiki read-write discovery records - t00_reduce_ttl (switchdc/oblivian@neodymium) |
[production] |
| 13:59 |
<START> |
- Reduce the TTL of all the MediaWiki read-write discovery records - t00_reduce_ttl (switchdc/oblivian@neodymium) |
[production] |
| 13:59 |
<END> |
(PASS) - Disabling puppet on selected hosts in codfw and eqiad - t00_disable_puppet (switchdc/oblivian@neodymium) |
[production] |
| 13:58 |
<START> |
- Disabling puppet on selected hosts in codfw and eqiad - t00_disable_puppet (switchdc/oblivian@neodymium) |
[production] |
| 13:16 |
<hashar> |
Restarting Jenkins |
[production] |
| 13:06 |
<marostegui> |
db1028: Increased /srv/ by 20G to clear the warning |
[production] |
| 11:59 |
<moritzm> |
rebooted kubernetes1002, not 1003 |
[production] |
| 11:59 |
<moritzm> |
rebooting kubernetes1003 for update to Linux 4.9 |
[production] |
| 11:39 |
<moritzm> |
rebooting kubernetes1001 for update to Linux 4.9 |
[production] |
| 11:37 |
<oblivian@naos> |
Synchronized wmf-config: Changing the read-only reason for the DC switchover (T164177) (duration: 01m 20s) |
[production] |