151-200 of 10000 results (26ms)
2017-05-02 ยง
16:51 <START> - Disabling puppet on selected hosts in eqiad and codfw - t00_disable_puppet (switchdc/volans@neodymium) [production]
16:51 <END> (PASS) - Reduce the TTL of all the MediaWiki read-write discovery records - t00_reduce_ttl (switchdc/volans@neodymium) [production]
16:50 <START> - Reduce the TTL of all the MediaWiki read-write discovery records - t00_reduce_ttl (switchdc/volans@neodymium) [production]
16:50 <END> (FAIL) - Reduce the TTL of all the MediaWiki read-write discovery records - t00_reduce_ttl (switchdc/volans@neodymium) [production]
16:50 <START> - Reduce the TTL of all the MediaWiki read-write discovery records - t00_reduce_ttl (switchdc/volans@neodymium) [production]
16:48 <ppchelko@naos> Started deploy [restbase/deploy@6adb0f2]: Summary endpoint enhancements. Restart after a check timeout [production]
16:47 <volans> testing (not dry-run) tasks for tomorrow's switchover in reverse mode eqiad->codfw [production]
16:43 <ppchelko@naos> Started deploy [restbase/deploy@6adb0f2]: Summary endpoint enhancements. Restart after a check fail [production]
16:42 <ppchelko@naos> Finished deploy [restbase/deploy@6adb0f2]: Summary endpoint enhancements (duration: 05m 47s) [production]
16:37 <ppchelko@naos> Started deploy [restbase/deploy@6adb0f2]: Summary endpoint enhancements [production]
16:36 <END> (PASS) - Wipe and warmup caches in codfw - t04_cache_wipe (switchdc/oblivian@neodymium) [production]
16:32 <END> (PASS) - Resync the redis for jobqueues in eqiad with the masters in codfw - t04_resync_redis (switchdc/oblivian@neodymium) [production]
16:32 <_joe_> message about cache warmup is wrong, it is being executed in eqiad [production]
16:29 <START> - Resync the redis for jobqueues in eqiad with the masters in codfw - t04_resync_redis (switchdc/oblivian@neodymium) [production]
16:29 <START> - Wipe and warmup caches in codfw - t04_cache_wipe (switchdc/oblivian@neodymium) [production]
16:29 <_joe_> testing (not dry-run) cache wipe/warmup and redis resync for the switchover codfw->eqiad [production]
16:25 <papaul> OS install on new db servers [production]
16:16 <elukey@naos> Synchronized wmf-config/ProductionServices.php: Replace Redis lock IPs after hw refresh (duration: 01m 16s) [production]
15:53 <oblivian@puppetmaster1001> conftool action : set/@read-only.yaml; selector: name=ReadOnly,scope=eqiad [production]
15:36 <ema> cache_misc: upgrade varnish to 4.1.6-1wm1 [production]
15:24 <_joe_> restarting confd in eqiad/esams to pick up the server change [production]
15:20 <godog> add 100G to graphite1003 and graphite2002 [production]
15:01 <elukey> stop and masked memcached on mc10[01-18].eqiad.wmnet [production]
14:35 <moritzm> rebooting rdb1007 for update to latest 4.4 kernel [production]
14:22 <moritzm> rebooting rdb1005 for update to latest 4.4 kernel [production]
13:52 <moritzm> rebooting rdb1003 for update to latest 4.4 kernel [production]
13:39 <moritzm> rebooting rdb1001 for update to latest 4.4 kernel [production]
13:26 <gehel> stopping load on elastic2020 - T149006 [production]
13:15 <ema> cache_maps: upgrade varnish to 4.1.6-1wm1 [production]
13:13 <gehel> load testing elastic2020 before putting it back in the cluster - T149006 [production]
13:03 <godog> rebuild mismounted FSes on ms-be1036 - T163673 [production]
12:22 <moritzm> rebooting rdb1008 for kernel update to Linux 4.9 [production]
12:19 <oblivian@puppetmaster1001> conftool action : set/pooled=yes; selector: cluster=pdf,name=ocg1001.eqiad.wmnet [production]
12:15 <_joe_> manually set ocg1001,3 to be redis slaves of ocg1002 [production]
11:47 <moritzm> rebooting rdb1006 for kernel update to Linux 4.9 [production]
11:37 <gehel> restart of relforge cluster to activate hebrew plugin [production]
11:30 <moritzm> rebooting rdb1004 for kernel update to Linux 4.9 [production]
11:23 <hashar> Restarting Nodepool [production]
11:23 <moritzm> downgraded python-jenkins on labnodepool1001 to 0.2.1 (0.4.11 is still broken with the new Jenkins LTS) [production]
11:06 <moritzm> rebooting rdb1002 for kernel update to Linux 4.9 [production]
10:51 <hashar> Restarting Nodepool with python-jenkins 0.4.11 [production]
10:50 <moritzm> upgrading python-jenkins on labnodepool1001 to 0.4.11 [production]
10:44 <akosiaris> create new ganeti nodegroup called row_A holding ganeti2005, ganeti2006. Renamed the default nodegroup to row_B. T164011 [production]
10:20 <elukey> restart ocg on ocg1002 (localhost:8000 - frontend - not reachable) [production]
10:12 <hashar> Upgrading Jenkins to 2.46.1 - T144106 [production]
10:11 <jynus> stopping replication on db1015 [production]
09:58 <END> (PASS) - Resync the redis for jobqueues in eqiad with the masters in codfw - t04_resync_redis (switchdc/oblivian@neodymium) [production]
09:56 <START> - Resync the redis for jobqueues in eqiad with the masters in codfw - t04_resync_redis (switchdc/oblivian@neodymium) [production]
09:55 <_joe_> testing pre-switchover the step to restart & resync redises in dc_to (eqiad) [production]
09:48 <jynus@naos> Synchronized wmf-config/db-codfw.php: Add db1097 (duration: 01m 00s) [production]