title: Spark job with large text file in gzip format
body: <p>I'm running a Spark job which is taking way too long time to process the input file. The input file is 6.8 GB in Gzip format and it contains 110 M lines of text. I know it's in Gzip format, so it's not splittable and only one executor will be used to read that file.</p>
  
  <p>As part of my debug process, I decided just to see how long does it take to convert that gzip file to parquet. My idea was that once I convert to parquet files and then if I run my original Spark job on that file, in that case it will use multiple executors and input file will be processed in parallel.</p>
  
  <p>But even the small job is taking a long time than I expected. Here is my code:</p>
  
  <pre><code>val input = sqlContext.read.text("input.gz")
  input.write.parquet("s3n://temp-output/")
  </code></pre>
  
  <p>When I extracted that file in my laptop (16 GB RAM), it took less than 2 minutes. When I run it on Spark cluster, my expectation was that it will take same or even less time since executor memory I was using is 58 GB. It took ~20 minutes.</p>
  
  <p>What am I missing here? I'm sorry if it sounds pretty amateur but I'm fairly new in Spark. </p>
  
  <p>What's the optimal way to run Spark job on gzip file? Assume I do not have option to create that file in other file format (bzip2, snappy, lzo).</p>