title: Redis: Spring Boot application requests keep failing while shutdown one of the redis master node
body: <p>&emsp;&emsp; Blocked by a redis issue these days, thanks for any suggestion in advance. Below are some details:</p>
  
  <blockquote>
    <p>Evn: Spring boot 2.0.3.RELEASE, Redis 3.0.6 cluster(3 master, 3 slave), </p>
    
    <p>Starter: spring-boot-starter-data-redis(defaul version with spring boot), which means the application will use letture as the redis client</p>
  </blockquote>
  
  <hr>
  
  <p>Error scenaro:</p>
  
  <ol>
  <li><p>Start the application and send some requests, everything goes fine</p></li>
  <li><p>Stop on the master node, the corresponding slave will takes about 20s to failover to be a master, which also goes fine.</p></li>
  <li><p>During the upper 20s(failover time) period, if keep sending the reuqest.</p>
  
  <ul>
  <li>In the 20s, requests fail, this is expected</li>
  <li>After the 20s(the slave becomes a master), requests still fail, this is <strong>unexpected</strong></li>
  </ul></li>
  <li><p>During the upper 20s(failover time) period, if no reuqest sent. After the slave becomes a master, the laster requests goes fine.</p></li>
  </ol>
  
  <blockquote>
    <p>no write operation during upper steps</p>
  </blockquote>
  
  <hr>
  
  <p>config:</p>
  
  <p><code>conf
      cache:
        type: redis
      redis:
        cluster: ip1:port(m),ip1:port(s),ip2:port(m),ip2:port(s),ip3:port(m),ip3:port(s)
        max-redirects: 3
      password: xxxx
      timeout: 1000
      pool:
        max-active: 500
        max-wait: 1500
  </code>   </p>
  
  <hr>
  
  <p>Code: just create a simple <code>CacheManager</code> bean</p>
  
  <p><code>java
  @Bean
  public CacheManager cacheManager(RedisConnectionFacotry redisConnectionFacotry){
     return new RedisCacheManager(RedisCacheWriter.nonLockingRedisCacheWriter(redisConnectionFacotry), 
            redisCacheConfig)   // set serializer and timeout
  }
  </code></p>
  
  <p>we use spring cache in the code with annotations like: <code>@CachePut</code> etc. The total data in redis is less than 10M and total volumn of redis is 2G.</p>
  
  <p>NEEDS YOUR HELP :) </p>