title: Apache Spark : TaskResultLost (result lost from block manager) Error On cluster
body: <p>I have a Spark standalone cluster with 3 slaves on Virtualbox. My code is on Java and it is working fine with my small input datasets which their inputs are around 100MB totally. </p>
  
  <p>I set my virtual machines RAM to be 16GB but when I was runnig my code on big input files (about 2GB) I get this error after hours of processing in my reduce part:</p>
  
  <pre><code>Job aborted due to stage failure: Total size of serialized results of 4 tasks (4.3GB) is bigger than spark.driver.maxResultSize`
  </code></pre>
  
  <p>I edited the <code>spark-defaults.conf</code> and assigned a higher amount (2GB and 4GB) for <code>spark.driver.maxResultSize</code>. It didn't help and the same error showed up.</p>
  
  <p>No I am trying 8GB of <code>spark.driver.maxResultSize</code> and my <code>spark.driver.memory</code> is also the same as RAM size (16GB). But I get this error:</p>
  
  <pre><code>TaskResultLost (result lost from block manager)
  </code></pre>
  
  <p>Any comments about this? I also include an image. </p>
  
  <p>I don't know if the problem is causing by the large size of <code>maxResultSize</code> or this is something with collections of RDDs in the code. I also provide the mapper part of the code for a better understanding.</p>
  
  <p><a href="https://i.stack.imgur.com/4gC0Y.jpg" rel="nofollow"><img src="https://i.stack.imgur.com/4gC0Y.jpg" alt="enter image description here"></a></p>
  
  <pre><code>JavaRDD&lt;Boolean[][][]&gt; fragPQ = uData.map(new Function&lt;String, Boolean[][][]&gt;() {
          public Boolean[][][] call(String s) {
              Boolean[][][] PQArr = new Boolean[2][][];
              PQArr[0] = new Boolean[11000][];
              PQArr[1] = new Boolean[11000][];
              for (int i = 0; i &lt; 11000; i++) {
                  PQArr[0][i] = new Boolean[11000];
                  PQArr[1][i] = new Boolean[11000];
                  for (int j = 0; j &lt; 11000; j++) {
                      PQArr[0][i][j] = true;
                      PQArr[1][i][j] = true;
  </code></pre>