title: Amazon AWS Client Timeout with lots of requests
body: <p>We have a Spring Boot application that stores multimedia files (up to 100 MB in size) in a S3 compatible cloud storage. The application receives these files via REST call or an AMQP message broker (RabbitMQ). </p>
  
  <p>Usually the load on the system is moderate so that there is no problem at all. However we encounter problems with accessing the S3 when there is heavy load on the system. Currently we are working around this issue with using a pool of 10 AmazonS3Clients that are assigned randomly to the calling process. This actually improves the issue but does not fix the problem. When the load is too high (meaning plenty of write and read operations) we encounter an exception of this sort:</p>
  
  <pre> com.amazonaws.AmazonClientException: Unable to execute HTTP request: connect timed out
      at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:299)
      at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:170)
      at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2648)
      at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1049)
      at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:924)</pre>
  
  <p>We're using the 1.3.8 version of the aws-java-sdk and cannot easily update to a newer version due to the region settings in the newer versions. The signing algorithm prevents us from accessing our buckets properly in the newest version.</p>
  
  <p>The implementation looks as follows:</p>
  
  <p>Initialization (at constructor level):</p>
  
  <pre><code>ClientConfiguration clientConfiguration = new ClientConfiguration();
  clientConfiguration.setConnectionTimeout(AWS_CONNECTION_TIMEOUT);
  clientConfiguration.setMaxConnections(AWS_MAX_CONNECTIONS);
  AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
  
  for (int i = 0; i &lt; AWS_MAX_CLIENTS; i++) {
       s3[i] = new AmazonS3Client(credentials, clientConfiguration);
       s3[i].setEndpoint(endpoint);       
  }
  </code></pre>
  
  <p>Put:</p>
  
  <pre><code>int i = getRandomClient();
  s3[i].putObject(bucketName, key, file);
  </code></pre>
  
  <p>Get:</p>
  
  <pre><code>ReadableByteChannel channel;
  try {
      int i = getRandomClient();
      S3Object object = s3[i].getObject(bucketName, addPrefix(fileId, prefix));
      S3ObjectInputStream stream = object.getObjectContent();
      channel = Channels.newChannel(stream);
  
      File file = File.createTempFile(fileId, "");
      try (WritableByteChannel outChannel = Channels.newChannel(new FileOutputStream(file))) {
          ByteBuffer buffer = ByteBuffer.allocate(8192);
          int read;
          while ((read = channel.read(buffer)) &gt; 0) {
              buffer.rewind();
              buffer.limit(read);
              while (read &gt; 0) {
                  read -= outChannel.write(buffer);
              }
              buffer.clear();
          }
          IOUtils.closeQuietly(stream);
          return file;
      }        
  }
  catch (AmazonClientException e) {
      if (!isMissingKey(e)) {
          throw new IOException(e);
      }
  }
  finally {
      if (channel != null) {
          channel.close();
      }
  }
  </code></pre>
  
  <p>It is pretty clear that the limited number of connections and clients is the bottleneck. There are plenty of ways how we could tweak the implementation to work properly. We could of course limit the number of consumers listening to the message broker. We could also increase the timeouts, number and connections of aws clients or limit the throughput in the service layer. However we're looking for a more sophisticated approach to handle things here.</p>
  
  <p>Is there any way to tell whether or not a designated client can currently be used or has too many open connections? Is there any way one could let the client wait for the next free connection?</p>