title: Best way to read huge files from S3 using Spring batch
body: <p>Requirement is to read files each &lt;4GB and push the data to some other location. I wrote my own implementation of getResources() by extending PathMatchingResourcePatternResolver class, so as to populate the resource array as below:</p>
  
  <pre><code>@Override
  public Resource[] getResources(String locationPattern) throws IOException {
      Set&lt;Resource&gt; resources = new HashSet&lt;Resource&gt;();
      AmazonS3 s3Client = streamingClient.getS3Client();
      for (S3ObjectSummary summary : S3Objects.withPrefix(s3Client, this.awsConfiguration.getS3Bucket(),
              this.awsConfiguration.getS3BucketKey())) {
          S3Object s3object = s3Client.getObject(new GetObjectRequest(this.awsConfiguration.getS3Bucket(), summary.getKey()));
          resources.add(new S3Resource(summary.getKey(), s3object.getObjectContent()));
      }
      return resources.toArray(new Resource[resources.size()]);
  }
  </code></pre>
  
  <p>As you can see, i read all resources and build a Resource array and return it. I think thats bad idea, rather than I prefer to pass the stream to somethread and continue to extract data and not to wait until all resources are populated in Resource array.</p>
  
  <p>Another draw back what i see is, holding the S3ObjectStream, which will block my request poolsize from S3.</p>
  
  <p>Is there any other way which I can use to build a reader ??</p>
  
  <p>Below is the data format, A1 has 1k + files, A2 has 1k+ files, each of size &lt; 4GB.</p>
  
  <p><code>
  &lt;bucket&gt;
    `--- A1/
          `--- file1.txt
          `---- file2.txt
     `---A2/
         `----- file3.txt
  ...........
  </code>
  Any help would be appreciated</p>