title: Reading multiple files from S3 in Spark by date period
body: <h1>Description</h1>
  
  <p>I have an application, which sends data to AWS Kinesis Firehose and this writes the data into my S3 bucket. Firehose uses "yyyy/MM/dd/HH" format to write the files.</p>
  
  <p>Like in this sample S3 path:</p>
  
  <pre><code>s3://mybucket/2016/07/29/12
  </code></pre>
  
  <p>Now I have a Spark application written in Scala, where I need to read data from a specific time period. I have start and end dates. The data is in JSON format and that's why I use <code>sqlContext.read.json()</code> not <code>sc.textFile()</code>.</p>
  
  <p>How can I read the data quickly and efficiently?</p>
  
  <h2>What have I tried?</h2>
  
  <ol>
  <li><p><strong>Wildcards</strong> - I can select the data from all hours of a specific date or all dates of a specific month, for example:</p>
  
  <pre><code>val df = sqlContext.read.json("s3://mybucket/2016/07/29/*")
  val df = sqlContext.read.json("s3://mybucket/2016/07/*/*")
  </code></pre>
  
  <p>But if I have to read data from the date period of a few days, for example 2016-07-29 - 2016-07-30 I cannot use the wildcard approach in the same way.</p>
  
  <p>Which brings me to my next point...</p></li>
  <li><strong>Using multiple paths</strong> or a CSV of directories as presented by <strong>samthebest</strong> in <a href="https://stackoverflow.com/questions/24029873/how-to-read-multiple-text-files-into-a-single-rdd">this</a> solution. It seems that separating directories with commas only works with <code>sc.textFile()</code> and not <code>sqlContext.read.json()</code>. </li>
  <li><p><strong>Union</strong> - A second solution from the previous link by <strong>cloud</strong> suggests to read each directory separately and then union them together. Although he suggests unioning RDD-s, there's an option to union DataFrames as well. If I generate the date strings from given date period manually, then I may create a path that does not exist and instead of ignoring it, the whole reading fails. Instead I could use AWS SDK and use the function <code>listObjects</code> from AmazonS3Client to get all the keys like in <strong>iMKanchwala</strong>'s solution from the previous link.</p>
  
  <p>The only problem is that my data is constantly changing. If <code>read.json()</code> function gets all the data as a single parameter, it reads all the necessary data and is smart enough to infer the json schema from the data. If I read 2 directories separately and their schemas don't match, then I think unioning these two dataframes becomes a problem.</p></li>
  <li><p><strong>Glob(?) syntax</strong> - <a href="https://stackoverflow.com/questions/31782763/apache-spark-using-regex-to-filter-input-files?noredirect=1&amp;lq=1">This</a> solution by <strong>nhahtdh</strong> is a little better than options <strong>1</strong> and <strong>2</strong> because they provide the option to specify dates and directories in more detail and as a single "path" so it works also with <code>read.json()</code>.</p>
  
  <p>But again, a familiar problem occurs about the missing directories. Let's say I want all the data from 20.07 to 30.07, I can declare it like this:</p>
  
  <pre><code>val df = sqlContext.read.json("s3://mybucket/2016/07/[20-30]/*")
  </code></pre>
  
  <p>But if I am missing data from let's say 25th of July, then the path <code>..16/07/25/</code> does not exist and the whole function fails.</p></li>
  </ol>
  
  <p>And obviously it gets more difficult when the requested period is for example 25.11.2015-12.02.2016, then I would need to programmatically (in my Scala script) create a string path something like this:</p>
  
  <pre><code>"s3://mybucket/{2015/11/[25-30],2015/12/*,2016/01/*,2016/02/[01-12]}/*"
  </code></pre>
  
  <p>And by creating it, I would neet to somehow be sure that these 25-30 and 01-12 intervals all have corresponding paths, if one is missing, it fails again. (Asterisk fortunately deals with missing directories, as it reads everything that exists)</p>
  
  <p><strong>How can I read all the necessary data from a single directory path all at once without the possibility of failing because of a missing directory between some date interval?</strong></p>