title: Reading from s3 bucket with thousands of s3 objects
body: <p>I have one s3 bucket with has 12,000 s3 objects. I want to read these s3 objects,in batches (say 50). I have set the number of max connections to 1000 (but we have 12,000 and its generally not a good practice to open these many connections at once when ultimately I am going to read only few at a time) I have multithreaded java code, which reads from these s3 objects, each s3 object is assigned to one thread. Here is the problem
  1. I want to read s3 objects in batches, say 50 at a time, but since we are reading only 50 objects from bucket and assigning S3ObjectInputStream to one thread, I am getting connection reset error for all other input streams have been waiting to actually start reading from s3 object</p>
  
  <blockquote>
    <p>java.net.SocketException: Connection reset</p>
  </blockquote>
  
  <p>What is the best way to approach this kind of problem. </p>
  
  <p>I have increased max number of connections to 1000, though I have 12k, how can i automate this, so that I dont have to split s3 bucket to 1000 objects at my end.</p>