title: 'Exception in thread "dispatcher-event-loop-0" java.lang.OutOfMemoryError: Java heap space ' error in Spark Scala code
body: <pre><code>val data = spark.read
      .text(filePath)
      .toDF("val")
      .withColumn("id", monotonically_increasing_id())
  
  
  
      val count = data.count()
  
  
  
      val header = data.where("id==1").collect().map(s =&gt; s.getString(0)).apply(0)
  
  
  
      val columns = header
      .replace("H|*|", "")
      .replace("|##|", "")
      .split("\\|\\*\\|")
  
  
      val structSchema = StructType(columns.map(s=&gt;StructField(s, StringType, true)))
  
  
  
      var correctData = data.where('id &gt; 1 &amp;&amp; 'id &lt; count-1).select("val")
      var dataString = correctData.collect().map(s =&gt; s.getString(0)).mkString("").replace("\\\n","").replace("\\\r","")
      var dataArr = dataString.split("\\|\\#\\#\\|").map(s =&gt;{ 
                                                            var arr = s.split("\\|\\*\\|")
                                                            while(arr.length &lt; columns.length) arr = arr :+ ""
                                                            RowFactory.create(arr:_*)
                                                           })
      val finalDF = spark.createDataFrame(sc.parallelize(dataArr),structSchema)
  
      display(finalDF)
  </code></pre>
  
  <p>This portion of code giving error:</p>
  
  <blockquote>
    <p>Exception in thread "dispatcher-event-loop-0" java.lang.OutOfMemoryError: Java heap space</p>
  </blockquote>
  
  <p>After hours of debugging mainly the part:</p>
  
  <pre><code>var dataArr = dataString.split("\\|\\#\\#\\|").map(s =&gt;{ 
                                                            var arr = s.split("\\|\\*\\|")
                                                            while(arr.length &lt; columns.length) arr = arr :+ ""
                                                            RowFactory.create(arr:_*)
                                                           })
      val finalDF = spark.createDataFrame(sc.parallelize(dataArr),structSchema)
  </code></pre>
  
  <p>causing the error.</p>
  
  <p>I changed the part as </p>
  
  <pre><code>var dataArr = dataString.split("\\|\\#\\#\\|").map(s =&gt;{
                                                            var arr = s.split("\\|\\*\\|")
                                                            while(arr.length &lt; columns.length) arr = arr :+ ""
                                                            RowFactory.create(arr:_*)
                                                           }).toList
    val finalDF = sqlContext.createDataFrame(sc.makeRDD(dataArr),structSchema)
  </code></pre>
  
  <p>But error remains same. What should I change to avoid this?</p>
  
  <p>When I ran this code is databricks spark cluster, particular job gives this Spark driver error:</p>
  
  <blockquote>
    <p>Job aborted due to stage failure: Serialized task 45:0 was 792585456 bytes, which exceeds max allowed: spark.rpc.message.maxSize (268435456 bytes). </p>
  </blockquote>
  
  <p>I added this portion of code:</p>
  
  <pre><code>spark.conf.set("spark.rpc.message.maxSize",Int.MaxValue)
  </code></pre>
  
  <p>but of no use.</p>