title: Spark: spark-csv takes too long
body: <p>I am trying to create a <code>DataFrame</code> from a CSV source that is on S3 on an EMR Spark cluster, using the Databricks <a href="https://github.com/databricks/spark-csv" rel="nofollow">spark-csv</a> package and the <a href="https://github.com/h2oai/h2o-2/wiki/Hacking-Airline-DataSet-with-H2O" rel="nofollow">flights dataset</a>:</p>
  
  <pre><code>from pyspark.sql import SQLContext
  sqlContext = SQLContext(sc)
  
  df = sqlContext.read.format('com.databricks.spark.csv').options(header='true').load('s3n://h2o-airlines-unpacked/allyears.csv')
  
  df.first()
  </code></pre>
  
  <p>This does not terminate on a cluster of 4 <code>m3.xlarge</code>s. I am looking for suggestions to create a <code>DataFrame</code> from a CSV file on S3 in PySpark. Alternatively, I have tried putting the file on HDFS and reading from HFDS as well, but that also does not terminate. The file is not overly large (12 GB).</p>