|
2017-05-03
ยง
|
| 19:18 |
<ppchelko@naos> |
Started deploy [restbase/deploy@76d909f]: Blacklist a title to fix cassandra OOMs T160759 |
[production] |
| 18:48 |
<papaul> |
db2084 - signing puppet certs, salt-key, initial run |
[production] |
| 18:48 |
<urandom> |
T160759: reducing tombstone threshold to 1000, restbase1014 |
[production] |
| 18:46 |
<urandom> |
T160759: reducing tombstone threshold to 1000, restbase1016 |
[production] |
| 18:39 |
<urandom> |
T160759: reducing tombstone threshold to 1000, restbase1013 |
[production] |
| 18:35 |
<urandom> |
restarting restbase1016-c |
[production] |
| 18:34 |
<urandom> |
restarting restbase1013-b |
[production] |
| 18:00 |
<bblack> |
restart cp2005 backend (lag) |
[production] |
| 17:33 |
<moritzm> |
uploaded openjdk-8 u131 to apt.wikimedia.org |
[production] |
| 17:14 |
<jynus@naos> |
Synchronized wmf-config/InitialiseSettings.php: Disable cognate- it is causing an outage on x1 (duration: 01m 06s) |
[production] |
| 16:30 |
<jynus@naos> |
Synchronized wmf-config/db-eqiad.php: Fine-tune per-server load to reduce db connection errors (duration: 01m 27s) |
[production] |
| 16:17 |
<mutante> |
install2002 / db2084 - reverting live hack, re-enabling puppet. db2084 doesnt even talk to DHCP, all other new db servers are fine, just this one out of 22 is not. seems to be actually broken NIC, cable was switched, switch config was checked too |
[production] |
| 16:08 |
<mutante> |
install2002 - temp stop puppet to debug dhcp issue of db2084 |
[production] |
| 15:13 |
<catrope@naos> |
Synchronized php-1.29.0-wmf.21/includes/logging/LogPager.php: Replace FORCE INDEX(ls_field_val) with IGNORE INDEX(ls_log_id) (https://gerrit.wikimedia.org/r/#/c/351653/ for T17441) (duration: 01m 14s) |
[production] |
| 15:09 |
<RoanKattouw> |
Live-hacked (cherry-picked) https://gerrit.wikimedia.org/r/#/c/351653/ onto naos and synced to mwdebug1002 for testing |
[production] |
| 14:54 |
<gehel> |
restart of elasticsearch on relforge |
[production] |
| 14:43 |
<END> |
(PASS) - Rolling restart of parsoid in codfw and eqiad - t09_restart_parsoid (switchdc/oblivian@neodymium) |
[production] |
| 14:27 |
<START> |
- Rolling restart of parsoid in codfw and eqiad - t09_restart_parsoid (switchdc/oblivian@neodymium) |
[production] |
| 14:26 |
<END> |
(PASS) - Update Tendril tree to start from the core DB masters in eqiad - t09_tendril (switchdc/oblivian@neodymium) |
[production] |
| 14:25 |
<START> |
- Update Tendril tree to start from the core DB masters in eqiad - t09_tendril (switchdc/oblivian@neodymium) |
[production] |
| 14:25 |
<godog> |
start swiftrepl on ms-fe1005 |
[production] |
| 14:24 |
<END> |
(PASS) - Start MediaWiki jobrunners, videoscalers and maintenance in eqiad - t09_start_maintenance (switchdc/oblivian@neodymium) |
[production] |
| 14:22 |
<START> |
- Start MediaWiki jobrunners, videoscalers and maintenance in eqiad - t09_start_maintenance (switchdc/oblivian@neodymium) |
[production] |
| 14:21 |
<END> |
(PASS) - Restore the TTL of all the MediaWiki read-write discovery records and cleanup confd stale files - t09_restore_ttl (switchdc/oblivian@neodymium) |
[production] |
| 14:21 |
<START> |
- Restore the TTL of all the MediaWiki read-write discovery records and cleanup confd stale files - t09_restore_ttl (switchdc/oblivian@neodymium) |
[production] |
| 14:20 |
<END> |
(PASS) - Set MediaWiki in read-write mode in eqiad (db-eqiad config already merged and git pulled) - t08_stop_mediawiki_readonly (switchdc/oblivian@neodymium) |
[production] |
| 14:20 |
<MediaWiki> |
read-only period ends at: 2017-05-03 14:20:28.286697 (switchdc/oblivian@neodymium) |
[production] |
| 14:20 |
<root@naos> |
Synchronized wmf-config/db-eqiad.php: Set MediaWiki in read-write mode in datacenter eqiad (duration: 00m 32s) |
[production] |
| 14:19 |
<START> |
- Set MediaWiki in read-write mode in eqiad (db-eqiad config already merged and git pulled) - t08_stop_mediawiki_readonly (switchdc/oblivian@neodymium) |
[production] |
| 14:19 |
<END> |
(PASS) - Set core DB masters in read-write mode in eqiad, ensure masters in codfw are read-only - t07_coredb_masters_readwrite (switchdc/oblivian@neodymium) |
[production] |
| 14:19 |
<START> |
- Set core DB masters in read-write mode in eqiad, ensure masters in codfw are read-only - t07_coredb_masters_readwrite (switchdc/oblivian@neodymium) |
[production] |
| 14:19 |
<END> |
(PASS) - Switch the Redis masters from codfw to eqiad and invert the replication - t06_redis (switchdc/oblivian@neodymium) |
[production] |
| 14:19 |
<START> |
- Switch the Redis masters from codfw to eqiad and invert the replication - t06_redis (switchdc/oblivian@neodymium) |
[production] |
| 14:18 |
<END> |
(PASS) - Switch traffic flow to the appservers from codfw to eqiad - t05_switch_traffic (switchdc/oblivian@neodymium) |
[production] |
| 14:17 |
<START> |
- Switch traffic flow to the appservers from codfw to eqiad - t05_switch_traffic (switchdc/oblivian@neodymium) |
[production] |
| 14:16 |
<END> |
(FAIL) - Switch MediaWiki master datacenter and read-write discovery records from codfw to eqiad - t05_switch_datacenter (switchdc/oblivian@neodymium) |
[production] |
| 14:16 |
<root@naos> |
Synchronized wmf-config/CommonSettings.php: Switch MediaWiki active datacenter to eqiad (duration: 00m 31s) |
[production] |
| 14:15 |
<START> |
- Switch MediaWiki master datacenter and read-write discovery records from codfw to eqiad - t05_switch_datacenter (switchdc/oblivian@neodymium) |
[production] |
| 14:15 |
<END> |
(PASS) - Wipe and warmup caches in eqiad - t04_cache_wipe (switchdc/oblivian@neodymium) |
[production] |
| 14:12 |
<elukey> |
restart kafka-mirror-main-eqiad_to_analytics.service on kafka1012 |
[production] |
| 14:12 |
<END> |
(PASS) - Resync the redis for jobqueues in eqiad with the masters in codfw - t04_resync_redis (switchdc/oblivian@neodymium) |
[production] |
| 14:09 |
<START> |
- Wipe and warmup caches in eqiad - t04_cache_wipe (switchdc/oblivian@neodymium) |
[production] |
| 14:08 |
<START> |
- Resync the redis for jobqueues in eqiad with the masters in codfw - t04_resync_redis (switchdc/oblivian@neodymium) |
[production] |
| 14:08 |
<END> |
(PASS) - Set core DB masters in read-only mode in codfw, ensure all masters are read-only - t03_coredb_masters_readonly (switchdc/oblivian@neodymium) |
[production] |
| 14:08 |
<START> |
- Set core DB masters in read-only mode in codfw, ensure all masters are read-only - t03_coredb_masters_readonly (switchdc/oblivian@neodymium) |
[production] |
| 14:08 |
<END> |
(PASS) - Set MediaWiki in read-only mode in codfw (db-codfw config already merged and git pulled) - t02_start_mediawiki_readonly (switchdc/oblivian@neodymium) |
[production] |
| 14:07 |
<root@naos> |
Synchronized wmf-config/db-codfw.php: Set MediaWiki in read-only mode in datacenter codfw (duration: 00m 45s) |
[production] |
| 14:07 |
<MediaWiki> |
read-only period starts at: 2017-05-03 14:07:08.261300 (switchdc/oblivian@neodymium) |
[production] |
| 14:07 |
<START> |
- Set MediaWiki in read-only mode in codfw (db-codfw config already merged and git pulled) - t02_start_mediawiki_readonly (switchdc/oblivian@neodymium) |
[production] |
| 14:06 |
<END> |
(PASS) - Stop MediaWiki jobrunners, videoscalers and cronjobs in codfw - t01_stop_maintenance (switchdc/oblivian@neodymium) |
[production] |