title: How to load a huge CSV to a Pyspark DataFrame?
body: <p>I'm trying to load a huge genomic dataset (2504 lines and 14848614 columns) to a PySpark DataFrame, but no success. I'm getting <code>java.lang.OutOfMemoryError: Java heap space</code>. I thought the main idea of using spark was exactly the independency of memory... (I'm newbie on it. Please, bear with me :)</p>
  
  <p>This is my code:</p>
  
  <pre><code>from pyspark.sql import SparkSession
  
  spark = SparkSession.builder.config("spark.driver.memory", "6G").getOrCreate()
  
  file_location = "1kGp3_chr3_6_10.raw"
  file_type = "csv"
  
  infer_schema = "true"
  first_row_is_header = "true"
  delimiter = "\t"
  max_cols = 15000000 # 14848614 variants loaded
  
  data = spark.read.format(file_type) \
    .option("inferSchema", infer_schema) \
    .option("header", first_row_is_header) \
    .option("sep", delimiter) \
    .option("maxColumns", max_cols) \
    .load(file_location)
  </code></pre>
  
  <p>I know we can set the <code>StorageLevel</code> by, for example <code>df.persist(StorageLevel.DISK_ONLY)</code>, but this is possible only after you successfully load the file to a DataFrame, isn't it? (not sure if I missing something)</p>
  
  <p>Here's the error: </p>
  
  <pre><code>...
  Py4JJavaError: An error occurred while calling o33.load.
  : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1, localhost, executor driver): java.lang.OutOfMemoryError: Java heap space
  ...
  </code></pre>
  
  <p>Thanks!</p>
  
  <hr>
  
  <p><strong>EDIT/UPDATE:</strong>
  I forgot to mention the size of the CSV: 70G.</p>
  
  <p>Here's another attempt which resulted in a different error: 
  I tried with a smaller dataset (2504 lines and 3992219 columns. File size: 19G), and increased memory to <code>"spark.driver.memory", "12G"</code>.
  After about 35 min running the load method, I got:</p>
  
  <pre><code>Py4JJavaError: An error occurred while calling o33.load.
  : org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 54 tasks (1033.1 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
      at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1889)
  </code></pre>