title: CloseableHttpClient.execute freezes once every few weeks despite timeouts
body: <p>We have a groovy singleton that uses PoolingHttpClientConnectionManager(httpclient:4.3.6) with a pool size of 200 to handle very high concurrent connections to a search service and processes the xml response.</p>
  
  <p>Despite having specified timeouts, it freezes about once a month but runs perfectly fine the rest of the time.</p>
  
  <p>The groovy singleton below. The method retrieveInputFromURL seems to block on client.execute(get);</p>
  
  
  
  
  
  <pre><code>@Singleton(strict=false)
  class StreamManagerUtil {
     // Instantiate once and cache for lifetime of Signleton class
  
     private static PoolingHttpClientConnectionManager connManager = new PoolingHttpClientConnectionManager();
  
     private static CloseableHttpClient client;
  
     private static final IdleConnectionMonitorThread staleMonitor = new IdleConnectionMonitorThread(connManager);
  
     private int warningLimit;
     private int readTimeout;
     private int connectionTimeout;
     private int connectionFetchTimeout;
  
     private int poolSize;
     private int routeSize;
  
     PropertyManager propertyManager  = PropertyManagerFactory.getInstance().getPropertyManager("sebe.properties")
  
     StreamManagerUtil() {
        // Initialize all instance variables in singleton from properties file
  
        readTimeout = 6
        connectionTimeout = 6
        connectionFetchTimeout =6
  
        // Pooling
        poolSize = 200
        routeSize = 50
  
        // Connection pool size and number of routes to cache
        connManager.setMaxTotal(poolSize);
        connManager.setDefaultMaxPerRoute(routeSize);
  
        // ConnectTimeout : time to establish connection with GSA
        // ConnectionRequestTimeout : time to get connection from pool
        // SocketTimeout : waiting for packets form GSA
  
        RequestConfig config = RequestConfig.custom()
        .setConnectTimeout(connectionTimeout * 1000)
        .setConnectionRequestTimeout(connectionFetchTimeout * 1000)
        .setSocketTimeout(readTimeout * 1000).build();
  
        // Keep alive for 5 seconds if server does not have keep alive header
        ConnectionKeepAliveStrategy myStrategy = new ConnectionKeepAliveStrategy() {
           @Override
           public long getKeepAliveDuration(HttpResponse response, HttpContext context) {
              HeaderElementIterator it = new BasicHeaderElementIterator
                 (response.headerIterator(HTTP.CONN_KEEP_ALIVE));
              while (it.hasNext()) {
                 HeaderElement he = it.nextElement();
                 String param = he.getName();
                 String value = he.getValue();
                 if (value != null &amp;&amp; param.equalsIgnoreCase
                    ("timeout")) {
                    return Long.parseLong(value) * 1000;
                 }
              }
              return 5 * 1000;
           }
        };
  
        // Close all connection older than 5 seconds. Run as separate thread.
        staleMonitor.start();
        staleMonitor.join(1000);
  
        client = HttpClients.custom().setDefaultRequestConfig(config).setKeepAliveStrategy(myStrategy).setConnectionManager(connManager).build();
     }     
  
     private retrieveInputFromURL (String categoryUrl, String xForwFor, boolean isXml) throws Exception {
  
        URL url = new URL( categoryUrl );
  
        GPathResult searchResponse = null
        InputStream inputStream = null
        HttpResponse response;
        HttpGet get;
        try {
           long startTime = System.nanoTime();
  
           get = new HttpGet(categoryUrl);
           response =  client.execute(get);
  
           int resCode = response.getStatusLine().getStatusCode();
  
           if (xForwFor != null) {
              get.setHeader("X-Forwarded-For", xForwFor)
           }
  
           if (resCode == HttpStatus.SC_OK) {
              if (isXml) {
                 extractXmlString(response)
              } else {
                 StringBuffer buffer = buildStringFromResponse(response)
                 return buffer.toString();
              }
           }
  
        }
        catch (Exception e)
        {
           throw e;
        }
        finally {
           // Release connection back to pool
           if (response != null) {
              EntityUtils.consume(response.getEntity());
           }
        }
  
     }
  
     private extractXmlString(HttpResponse response) {
        InputStream inputStream = response.getEntity().getContent()
  
        XmlSlurper slurper = new XmlSlurper()
        slurper.setFeature("http://xml.org/sax/features/validation", false)
        slurper.setFeature("http://apache.org/xml/features/disallow-doctype-decl", false)
        slurper.setFeature("http://apache.org/xml/features/nonvalidating/load-dtd-grammar", false)
        slurper.setFeature("http://apache.org/xml/features/nonvalidating/load-external-dtd", false)
  
        return slurper.parse(inputStream)
     }
  
     private StringBuffer buildStringFromResponse(HttpResponse response) {
        StringBuffer buffer= new StringBuffer();
        BufferedReader rd = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
        String line = "";
        while ((line = rd.readLine()) != null) {
           buffer.append(line);
           System.out.println(line);
        }
        return buffer
     }
  </code></pre>
  
  
  
  
  
  
  
  <pre><code>public class IdleConnectionMonitorThread extends Thread {
  
      private final HttpClientConnectionManager connMgr;
      private volatile boolean shutdown;
  
      public IdleConnectionMonitorThread
        (PoolingHttpClientConnectionManager connMgr) {
          super();
          this.connMgr = connMgr;
      }
  
      @Override
      public void run() {
          try {
              while (!shutdown) {
                  synchronized (this) {
                      wait(5000);
                      connMgr.closeExpiredConnections();
                      connMgr.closeIdleConnections(10, TimeUnit.SECONDS);
                  }
              }
          } catch (InterruptedException ex) {
              // Ignore
          }
      }
      public void shutdown() {
          shutdown = true;
          synchronized (this) {
              notifyAll();
          }
      }
  }
  </code></pre>
  
  
  
  <p>I also found found this in the log leading me to believe it happened on waiting for response data </p>
  
  <p>
  </p>
  
  <p>java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)</p>
  
  
  
  <h2>Findings thus far:</h2>
  
  <ul>
  <li>We are using java 1.8u25. There is an open issue on a similar scenario
  <a href="https://bugs.openjdk.java.net/browse/JDK-8075484">https://bugs.openjdk.java.net/browse/JDK-8075484</a></li>
  <li>HttpClient had a similar report <a href="https://issues.apache.org/jira/browse/HTTPCLIENT-1589">https://issues.apache.org/jira/browse/HTTPCLIENT-1589</a> but this was fixed in 
  the 4.3.6 version we are using</li>
  </ul>
  
  <h2>Questions</h2>
  
  <ul>
  <li>Can this be a synchronisation issue? From my understanding even though the singleton is accessed by multiple threads, the only shared data is the cached CloseableHttpClient</li>
  <li>Is there anything else fundamentally wrong with this code,approach that may be causing this behaviour?</li>
  </ul>