|
2017-05-03
ยง
|
| 14:17 |
<START> |
- Switch traffic flow to the appservers from codfw to eqiad - t05_switch_traffic (switchdc/oblivian@neodymium) |
[production] |
| 14:16 |
<END> |
(FAIL) - Switch MediaWiki master datacenter and read-write discovery records from codfw to eqiad - t05_switch_datacenter (switchdc/oblivian@neodymium) |
[production] |
| 14:16 |
<root@naos> |
Synchronized wmf-config/CommonSettings.php: Switch MediaWiki active datacenter to eqiad (duration: 00m 31s) |
[production] |
| 14:15 |
<START> |
- Switch MediaWiki master datacenter and read-write discovery records from codfw to eqiad - t05_switch_datacenter (switchdc/oblivian@neodymium) |
[production] |
| 14:15 |
<END> |
(PASS) - Wipe and warmup caches in eqiad - t04_cache_wipe (switchdc/oblivian@neodymium) |
[production] |
| 14:12 |
<elukey> |
restart kafka-mirror-main-eqiad_to_analytics.service on kafka1012 |
[production] |
| 14:12 |
<END> |
(PASS) - Resync the redis for jobqueues in eqiad with the masters in codfw - t04_resync_redis (switchdc/oblivian@neodymium) |
[production] |
| 14:09 |
<START> |
- Wipe and warmup caches in eqiad - t04_cache_wipe (switchdc/oblivian@neodymium) |
[production] |
| 14:08 |
<START> |
- Resync the redis for jobqueues in eqiad with the masters in codfw - t04_resync_redis (switchdc/oblivian@neodymium) |
[production] |
| 14:08 |
<END> |
(PASS) - Set core DB masters in read-only mode in codfw, ensure all masters are read-only - t03_coredb_masters_readonly (switchdc/oblivian@neodymium) |
[production] |
| 14:08 |
<START> |
- Set core DB masters in read-only mode in codfw, ensure all masters are read-only - t03_coredb_masters_readonly (switchdc/oblivian@neodymium) |
[production] |
| 14:08 |
<END> |
(PASS) - Set MediaWiki in read-only mode in codfw (db-codfw config already merged and git pulled) - t02_start_mediawiki_readonly (switchdc/oblivian@neodymium) |
[production] |
| 14:07 |
<root@naos> |
Synchronized wmf-config/db-codfw.php: Set MediaWiki in read-only mode in datacenter codfw (duration: 00m 45s) |
[production] |
| 14:07 |
<MediaWiki> |
read-only period starts at: 2017-05-03 14:07:08.261300 (switchdc/oblivian@neodymium) |
[production] |
| 14:07 |
<START> |
- Set MediaWiki in read-only mode in codfw (db-codfw config already merged and git pulled) - t02_start_mediawiki_readonly (switchdc/oblivian@neodymium) |
[production] |
| 14:06 |
<END> |
(PASS) - Stop MediaWiki jobrunners, videoscalers and cronjobs in codfw - t01_stop_maintenance (switchdc/oblivian@neodymium) |
[production] |
| 14:01 |
<START> |
- Stop MediaWiki jobrunners, videoscalers and cronjobs in codfw - t01_stop_maintenance (switchdc/oblivian@neodymium) |
[production] |
| 14:00 |
<godog> |
stop swiftrepl on ms-fe1005 |
[production] |
| 13:59 |
<END> |
(PASS) - Reduce the TTL of all the MediaWiki read-write discovery records - t00_reduce_ttl (switchdc/oblivian@neodymium) |
[production] |
| 13:59 |
<START> |
- Reduce the TTL of all the MediaWiki read-write discovery records - t00_reduce_ttl (switchdc/oblivian@neodymium) |
[production] |
| 13:59 |
<END> |
(PASS) - Disabling puppet on selected hosts in codfw and eqiad - t00_disable_puppet (switchdc/oblivian@neodymium) |
[production] |
| 13:58 |
<START> |
- Disabling puppet on selected hosts in codfw and eqiad - t00_disable_puppet (switchdc/oblivian@neodymium) |
[production] |
| 13:16 |
<hashar> |
Restarting Jenkins |
[production] |
| 13:06 |
<marostegui> |
db1028: Increased /srv/ by 20G to clear the warning |
[production] |
| 11:59 |
<moritzm> |
rebooted kubernetes1002, not 1003 |
[production] |
| 11:59 |
<moritzm> |
rebooting kubernetes1003 for update to Linux 4.9 |
[production] |
| 11:39 |
<moritzm> |
rebooting kubernetes1001 for update to Linux 4.9 |
[production] |
| 11:37 |
<oblivian@naos> |
Synchronized wmf-config: Changing the read-only reason for the DC switchover (T164177) (duration: 01m 20s) |
[production] |
| 11:25 |
<moritzm> |
uploaded nodepool 0.1.1+wmf7 to apt.wikimedia.org |
[production] |
| 11:23 |
<hashar> |
Upgrading Jenkins 2.46.1 -> 2.46.2 - T144106 |
[production] |
| 11:16 |
<jynus> |
restarting replication on s*, and x1 eqiad -> codfw |
[production] |
| 11:02 |
<hashar> |
Restarting Nodepool |
[production] |
| 10:58 |
<moritzm> |
upgrading nodepool on labnodepool1001 to a package including https://gerrit.wikimedia.org/r/351608 |
[production] |
| 10:18 |
<END> |
(PASS) - Switch MediaWiki master datacenter and read-write discovery records from eqiad to codfw - t05_switch_datacenter (switchdc/oblivian@neodymium) |
[production] |
| 10:17 |
<START> |
- Switch MediaWiki master datacenter and read-write discovery records from eqiad to codfw - t05_switch_datacenter (switchdc/oblivian@neodymium) |
[production] |
| 10:14 |
<END> |
(PASS) - Set MediaWiki in read-write mode in codfw (db-codfw config already merged and git pulled) - t08_stop_mediawiki_readonly (switchdc/oblivian@neodymium) |
[production] |
| 10:14 |
<START> |
- Set MediaWiki in read-write mode in codfw (db-codfw config already merged and git pulled) - t08_stop_mediawiki_readonly (switchdc/oblivian@neodymium) |
[production] |
| 10:14 |
<END> |
(PASS) - Set MediaWiki in read-only mode in eqiad (db-eqiad config already merged and git pulled) - t02_start_mediawiki_readonly (switchdc/oblivian@neodymium) |
[production] |
| 10:13 |
<START> |
- Set MediaWiki in read-only mode in eqiad (db-eqiad config already merged and git pulled) - t02_start_mediawiki_readonly (switchdc/oblivian@neodymium) |
[production] |
| 10:13 |
<_joe_> |
testing reverted steps of switchdc, non-dry-run --dc-from eqiad --dc-to codfw (should be noop) |
[production] |
| 10:05 |
<moritzm> |
installing icu security updates on trusty (jessie already fixed) |
[production] |
| 09:50 |
<marostegui> |
Restart db1097 to change its binlog to STATEMENT - T155099 |
[production] |
| 09:19 |
<elukey> |
reboot mc[1019-1036].eqiad.wmnet for kernel upgrades |
[production] |
| 09:18 |
<moritzm> |
rebooting restbase1018 for update to Linux 4.9 |
[production] |
| 09:05 |
<godog> |
rebuild mismounted FSes on ms-be1035 - T163673 |
[production] |
| 08:53 |
<_joe_> |
rebooting restbase1018 T163280 |
[production] |
| 08:24 |
<_joe_> |
deactivating restbase1018-vg for RAID failover and rebuild T163280 |
[production] |
| 08:01 |
<hashar> |
Rolling back Jenkins 2.46.2 -> 2.46.1 - T144106 |
[production] |
| 07:53 |
<hashar> |
Upgrading Jenkins 2.46.1 -> 2.46.2 - T144106 |
[production] |
| 07:42 |
<_joe_> |
rebuilding RAIDs on restbase1018 T163280 |
[production] |