title: Spark DataFrame save as parquet - out of memory
body: <p>I am using spark to read a file from s3, then I load it to a dataframe and then i try to write it to hdfs as parquet.
  The thing is that when the file is big (65G), and for some reason I get out of memory...in any case I have no idea why I get the outof memory because it looks like the data is partitioned pretty well.</p>
  
  <p>this is a sampel of my code:</p>
  
  <pre><code>val records = gzCsvFile.filter { x =&gt; x.length == 31 }.map { x =&gt; 
  
     var d:Date = Date.valueOf(x(0))
  
        //var z = new GregorianCalendar();z.getWeekYear   
  
      var week = (1900+d.getYear )* 1000 + d.getMonth()*10 + Math.ceil(d.getDate()/7.0).toInt 
      Row(d, Timestamp.valueOf(x(1)), toLong( x(2)), toLong(x(3)), toLong(x(4)), toLong(x(5)), toLong(x(6)), toLong(x(7)), toLong(x(8)), toLong(x(9)), toLong(x(10)), toLong(x(11)), toLong(x(12)), toLong(x(13)), toLong(x(14)), toLong(x(15)), toLong(x(16)), toLong(x(17)), toLong(x(18)), toLong(x(19)), toLong(x(20)), toLong(x(21)), toLong(x(22)), toLong(x(23)), toLong(x(24)), toLong(x(25)), x(26).trim(), toLong(x(27)), toLong(x(28)), toLong(x(29)), toInt(x(30)),  week)
  }
  var cubeDF = sqlContext.createDataFrame(records, cubeSchema)
  cubeDF.write.mode(SaveMode.Overwrite).partitionBy("CREATION_DATE","COUNTRY_ID","CHANNEL_ID" ).parquet(cubeParquetUrl)
  </code></pre>
  
  <p>does anyone have any idea what is going on?</p>