|
2017-05-02
ยง
|
| 16:57 |
<MediaWiki> |
read-only period starts at: 2017-05-02 16:57:37.952132 (switchdc/volans@neodymium) |
[production] |
| 16:57 |
<START> |
- Set MediaWiki in read-only mode in eqiad - t02_start_mediawiki_readonly (switchdc/volans@neodymium) |
[production] |
| 16:56 |
<ppchelko@naos> |
Finished deploy [restbase/deploy@6adb0f2]: Summary endpoint enhancements. Restart after a check timeout (duration: 07m 56s) |
[production] |
| 16:53 |
<END> |
(FAIL) - Stop MediaWiki jobrunners, videoscalers and cronjobs in eqiad - t01_stop_maintenance (switchdc/volans@neodymium) |
[production] |
| 16:53 |
<START> |
- Stop MediaWiki jobrunners, videoscalers and cronjobs in eqiad - t01_stop_maintenance (switchdc/volans@neodymium) |
[production] |
| 16:52 |
<END> |
(PASS) - Disabling puppet on selected hosts in eqiad and codfw - t00_disable_puppet (switchdc/volans@neodymium) |
[production] |
| 16:51 |
<START> |
- Disabling puppet on selected hosts in eqiad and codfw - t00_disable_puppet (switchdc/volans@neodymium) |
[production] |
| 16:51 |
<END> |
(PASS) - Reduce the TTL of all the MediaWiki read-write discovery records - t00_reduce_ttl (switchdc/volans@neodymium) |
[production] |
| 16:50 |
<START> |
- Reduce the TTL of all the MediaWiki read-write discovery records - t00_reduce_ttl (switchdc/volans@neodymium) |
[production] |
| 16:50 |
<END> |
(FAIL) - Reduce the TTL of all the MediaWiki read-write discovery records - t00_reduce_ttl (switchdc/volans@neodymium) |
[production] |
| 16:50 |
<START> |
- Reduce the TTL of all the MediaWiki read-write discovery records - t00_reduce_ttl (switchdc/volans@neodymium) |
[production] |
| 16:48 |
<ppchelko@naos> |
Started deploy [restbase/deploy@6adb0f2]: Summary endpoint enhancements. Restart after a check timeout |
[production] |
| 16:47 |
<volans> |
testing (not dry-run) tasks for tomorrow's switchover in reverse mode eqiad->codfw |
[production] |
| 16:43 |
<ppchelko@naos> |
Started deploy [restbase/deploy@6adb0f2]: Summary endpoint enhancements. Restart after a check fail |
[production] |
| 16:42 |
<ppchelko@naos> |
Finished deploy [restbase/deploy@6adb0f2]: Summary endpoint enhancements (duration: 05m 47s) |
[production] |
| 16:37 |
<ppchelko@naos> |
Started deploy [restbase/deploy@6adb0f2]: Summary endpoint enhancements |
[production] |
| 16:36 |
<END> |
(PASS) - Wipe and warmup caches in codfw - t04_cache_wipe (switchdc/oblivian@neodymium) |
[production] |
| 16:32 |
<END> |
(PASS) - Resync the redis for jobqueues in eqiad with the masters in codfw - t04_resync_redis (switchdc/oblivian@neodymium) |
[production] |
| 16:32 |
<_joe_> |
message about cache warmup is wrong, it is being executed in eqiad |
[production] |
| 16:29 |
<START> |
- Resync the redis for jobqueues in eqiad with the masters in codfw - t04_resync_redis (switchdc/oblivian@neodymium) |
[production] |
| 16:29 |
<START> |
- Wipe and warmup caches in codfw - t04_cache_wipe (switchdc/oblivian@neodymium) |
[production] |
| 16:29 |
<_joe_> |
testing (not dry-run) cache wipe/warmup and redis resync for the switchover codfw->eqiad |
[production] |
| 16:25 |
<papaul> |
OS install on new db servers |
[production] |
| 16:16 |
<elukey@naos> |
Synchronized wmf-config/ProductionServices.php: Replace Redis lock IPs after hw refresh (duration: 01m 16s) |
[production] |
| 15:53 |
<oblivian@puppetmaster1001> |
conftool action : set/@read-only.yaml; selector: name=ReadOnly,scope=eqiad |
[production] |
| 15:36 |
<ema> |
cache_misc: upgrade varnish to 4.1.6-1wm1 |
[production] |
| 15:24 |
<_joe_> |
restarting confd in eqiad/esams to pick up the server change |
[production] |
| 15:20 |
<godog> |
add 100G to graphite1003 and graphite2002 |
[production] |
| 15:01 |
<elukey> |
stop and masked memcached on mc10[01-18].eqiad.wmnet |
[production] |
| 14:35 |
<moritzm> |
rebooting rdb1007 for update to latest 4.4 kernel |
[production] |
| 14:22 |
<moritzm> |
rebooting rdb1005 for update to latest 4.4 kernel |
[production] |
| 13:52 |
<moritzm> |
rebooting rdb1003 for update to latest 4.4 kernel |
[production] |
| 13:39 |
<moritzm> |
rebooting rdb1001 for update to latest 4.4 kernel |
[production] |
| 13:26 |
<gehel> |
stopping load on elastic2020 - T149006 |
[production] |
| 13:15 |
<ema> |
cache_maps: upgrade varnish to 4.1.6-1wm1 |
[production] |
| 13:13 |
<gehel> |
load testing elastic2020 before putting it back in the cluster - T149006 |
[production] |
| 13:03 |
<godog> |
rebuild mismounted FSes on ms-be1036 - T163673 |
[production] |
| 12:22 |
<moritzm> |
rebooting rdb1008 for kernel update to Linux 4.9 |
[production] |
| 12:19 |
<oblivian@puppetmaster1001> |
conftool action : set/pooled=yes; selector: cluster=pdf,name=ocg1001.eqiad.wmnet |
[production] |
| 12:15 |
<_joe_> |
manually set ocg1001,3 to be redis slaves of ocg1002 |
[production] |
| 11:47 |
<moritzm> |
rebooting rdb1006 for kernel update to Linux 4.9 |
[production] |
| 11:37 |
<gehel> |
restart of relforge cluster to activate hebrew plugin |
[production] |
| 11:30 |
<moritzm> |
rebooting rdb1004 for kernel update to Linux 4.9 |
[production] |
| 11:23 |
<hashar> |
Restarting Nodepool |
[production] |
| 11:23 |
<moritzm> |
downgraded python-jenkins on labnodepool1001 to 0.2.1 (0.4.11 is still broken with the new Jenkins LTS) |
[production] |
| 11:06 |
<moritzm> |
rebooting rdb1002 for kernel update to Linux 4.9 |
[production] |
| 10:51 |
<hashar> |
Restarting Nodepool with python-jenkins 0.4.11 |
[production] |
| 10:50 |
<moritzm> |
upgrading python-jenkins on labnodepool1001 to 0.4.11 |
[production] |
| 10:44 |
<akosiaris> |
create new ganeti nodegroup called row_A holding ganeti2005, ganeti2006. Renamed the default nodegroup to row_B. T164011 |
[production] |
| 10:20 |
<elukey> |
restart ocg on ocg1002 (localhost:8000 - frontend - not reachable) |
[production] |