title: Apache Spark Effects of Driver Memory, Executor Memory, Driver Memory Overhead and Executor Memory Overhead on success of job runs
body: <p>I am doing some memory tuning on my Spark job on YARN and I notice different settings would give different results and affect the outcome of the Spark job run. However, I am confused and do not understand completely why it happens and would appreciate if someone can provide me with some guidance and explanation. </p>
  
  <p>I will provide some background information and post my questions and describe the cases that I have experienced after them below.</p>
  
  <p><strong>My environment setting were as below:</strong></p>
  
  <ul>
  <li>Memory 20G, 20 VCores per node (3 nodes in total)</li>
  <li>Hadoop 2.6.0</li>
  <li>Spark 1.4.0</li>
  </ul>
  
  <p>My code recursively filters an RDD to make it smaller (removing examples as part of an algorithm), then does mapToPair and collect to gather the results and save them within a list.</p>
  
  <h1>Questions</h1>
  
  <ol>
  <li><p>Why is a different error thrown and the job runs longer (for the second case) between the first and second case with only the executor memory being increased? Are the two errors linked in some way?</p></li>
  <li><p>Both the third and fourth case succeeds and I understand that it is because I am giving more memory which solves the memory problems. However, in the third case,</p></li>
  </ol>
  
  <blockquote>
    <p>spark.driver.memory + spark.yarn.driver.memoryOverhead = the memory
    that YARN will create a JVM</p>
    
    <p>= 11g + (driverMemory * 0.07, with minimum of 384m) 
    = 11g + 1.154g
    = 12.154g</p>
  </blockquote>
  
  <p>So, from the formula, I can see that my job requires <em>MEMORY_TOTAL of around 12.154g</em> to run successfully which explains why I need more than 10g for the driver memory setting.</p>
  
  <p>But for the fourth case, </p>
  
  <blockquote>
    <p>spark.driver.memory + spark.yarn.driver.memoryOverhead = the memory
    that YARN will create a JVM</p>
    
    <p>= 2 + (driverMemory * 0.07, with minimum of 384m) 
    = 2g + 0.524g
    = 2.524g</p>
  </blockquote>
  
  <p>It seems that just by increasing the memory overhead by a small amount of 1024(1g) it leads to the successful run of the job with driver memory of only 2g and the <em>MEMORY_TOTAL is only 2.524g</em>! Whereas without the overhead configuration, driver memory less than 11g fails but it doesn't make sense from the formula which is why I am confused.</p>
  
  <p>Why increasing the memory overhead (for both driver and executor) allows my job to complete successfully with a lower <em>MEMORY_TOTAL (12.154g vs 2.524g)</em>? Is there some other internal things at work here that I am missing?</p>
  
  <h2> First Case </h2> 
  
  <p><code>/bin/spark-submit --class &lt;class name&gt; --master yarn-cluster --driver-memory 7g --executor-memory 1g --num-executors 3 --executor-cores 1 --jars &lt;jar file&gt;</code></p>
  
  <p>If I run my program with any driver memory less than 11g, I will get the error below which is the SparkContext being stopped or a similar error which is a method being called on a stopped SparkContext. From what I have gathered, this is related to memory not being enough.</p>
  
  <p><a href="https://i.stack.imgur.com/EKxQD.png" rel="noreferrer"><img src="https://i.stack.imgur.com/EKxQD.png" alt="DriverMemory-7g_ExecutorMemory-1g"></a></p>
  
  <h2>Second Case</h2> 
  
  <p><code>/bin/spark-submit --class &lt;class name&gt; --master yarn-cluster --driver-memory 7g --executor-memory 3g --num-executors 3 --executor-cores 1 --jars &lt;jar file&gt;</code></p>
  
  <p>If I run the program with the same driver memory but higher executor memory, the job runs longer (about 3-4 minutes) than the first case and then it will encounter a different error from earlier which is a Container requesting/using more memory than allowed and is being killed because of that. Although I find it weird since the executor memory is increased and this error occurs instead of the error in the first case.</p>
  
  <p><a href="https://i.stack.imgur.com/tr24f.png" rel="noreferrer"><img src="https://i.stack.imgur.com/tr24f.png" alt="DriverMemory-7g_ExecutorMemory-3g"></a></p>
  
  <h2>Third Case</h2>  
  
  <p><code>/bin/spark-submit --class &lt;class name&gt; --master yarn-cluster --driver-memory 11g --executor-memory 1g --num-executors 3 --executor-cores 1 --jars &lt;jar file&gt;</code></p>
  
  <p>Any setting with driver memory greater than 10g will lead to the job being able to run successfully.</p>
  
  <h2>Fourth Case</h2> 
  
  <p><code>/bin/spark-submit --class &lt;class name&gt; --master yarn-cluster --driver-memory 2g --executor-memory 1g --conf spark.yarn.executor.memoryOverhead=1024 --conf spark.yarn.driver.memoryOverhead=1024 --num-executors 3 --executor-cores 1 --jars &lt;jar file&gt;</code></p>
  
  <p>The job will run successfully with this setting (driver memory 2g and executor memory 1g but increasing the driver memory overhead(1g) and the executor memory overhead(1g).</p>
  
  <p>Any help will be appreciated and would really help with my understanding of Spark. Thanks in advance.</p>