title: Reading multiple files from S3 in parallel (Spark, Java)
body: <p>I saw a few discussions on this but couldn't quite understand the right solution:
  I want to load a couple hundred files from S3 into an RDD. Here is how I'm doing it now:</p>
  
  <pre><code>ObjectListing objectListing = s3.listObjects(new ListObjectsRequest().
                  withBucketName(...).
                  withPrefix(...));
  List&lt;String&gt; keys = new LinkedList&lt;&gt;();
  objectListing.getObjectSummaries().forEach(summery -&gt; keys.add(summery.getKey())); // repeat while objectListing.isTruncated()
  
  JavaRDD&lt;String&gt; events = sc.parallelize(keys).flatMap(new ReadFromS3Function(clusterProps));
  </code></pre>
  
  <p>The <code>ReadFromS3Function</code> does the actual reading using the <code>AmazonS3</code> client:</p>
  
  <pre><code>    public Iterator&lt;String&gt; call(String s) throws Exception {
          AmazonS3 s3Client = getAmazonS3Client(properties);
          S3Object object = s3Client.getObject(new GetObjectRequest(...));
          InputStream is = object.getObjectContent();
          List&lt;String&gt; lines = new LinkedList&lt;&gt;();
          String str;
          try {
              BufferedReader reader = new BufferedReader(new InputStreamReader(is));
              if (is != null) {
                  while ((str = reader.readLine()) != null) {
                      lines.add(str);
                  }
              } else {
                  ...
              }
          } finally {
              ...
          }
          return lines.iterator();
  </code></pre>
  
  <p>I  kind of "translated" this from answers I saw for the same question in Scala. I think it's also possible to pass the entire list of paths to <code>sc.textFile(...)</code>, but I'm not sure which is the best-practice way.</p>