title: Reading in multiple files compressed in tar.gz archive into Spark
body: <p>I'm trying to create a Spark RDD from several json files compressed into a tar.
  For example, I have 3 files </p>
  
  <pre><code>file1.json
  file2.json
  file3.json
  </code></pre>
  
  <p>And these are contained in <code>archive.tar.gz</code>.</p>
  
  <p>I want to create a dataframe from the json files. The problem is that Spark is not reading in the json files correctly. Creating an RDD using <code>sqlContext.read.json("archive.tar.gz")</code> or <code>sc.textFile("archive.tar.gz")</code> results in garbled/extra output. </p>
  
  <p>Is there some way to handle gzipped archives containing multiple files in Spark?</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>Using the method given in the answer to <a href="https://stackoverflow.com/questions/36604145/read-whole-text-files-from-a-compression-in-spark">Read whole text files from a compression in Spark</a> I was able to get things running, but this method does not seem to be suitable for large tar.gz archives (>200 mb compressed) as the application chokes up on large archive sizes. As some of the archives I'm dealing with reach sizes upto <strong>2 GB</strong> after compression I'm wondering if there is some efficient way to deal with the problem.</p>
  
  <p>I'm trying to avoid extracting the archives and then merging the files together as this would be time consuming.</p>