title: Spark job in Dataproc dynamic vs static allocation
body: <p>I have a Dataproc cluster:</p>
  
  <p>master - 6cores| 32g</p>
  
  <p>worker{0-7} - 6cores| 32g</p>
  
  <p>Maximum allocation: memory:24576, vCores:6</p>
  
  <p>Have two spark-streaming jobs to submit, one after another</p>
  
  <p>In the first place, I tried to submit with default configurations <code>spark.dynamicAllocation.enabled=true</code></p>
  
  <p>In 30% of cases, I saw that the first job caught almost all available memory and the second was queued and waited for resources for ages. (This is a streaming job which took a small portion of resources every batch ).</p>
  
  <p>My second try was to change a dynamic allocation. I submitted the same two jobs with identical configurations: </p>
  
  <pre><code>spark.dynamicAllocation.enabled=false
  spark.executor.memory=12g
  spark.executor.cores=3
  spark.executor.instances=6
  spark.driver.memory=8g
  </code></pre>
  
  <p>Surprisingly in Yarn UI I saw:</p>
  
  <p><strong>7 Running Containers with 84g Memory allocation</strong> for the first job.</p>
  
  <p><strong>3 Running Containers with 36g Memory allocation and 72g Reserved Memory</strong> for the second job</p>
  
  <p><em>In Spark UI</em> there are 6 executors and driver for the first job and 2 executors and driver for the second job</p>
  
  <p>After retrying(deleting previous jobs and submitting the same jobs) without dynamic allocation and same configurations, I got a totally different result:</p>
  
  <p><strong>5 containers 59g Memory allocation</strong> for both jobs and <strong>71g Reserved Memory</strong> for the second job. In spark UI I see 4 executors and driver in both cases.</p>
  
  <p>I have a couple of questions: </p>
  
  <ol>
  <li><strong>If dynamicAllocation=false</strong>, why the number of yarn containers is 
  different from the number of executors? (Firstly I thought that 
  additional yarn container is a driver, but it differs in memory.)</li>
  <li><strong>If dynamicAllocation=false</strong>, Why Yarn doesn't create containers by my 
  exact requirements- 6 containers(spark executors) for both jobs. Why two different attempts with the same configuration lead to different results?</li>
  <li><strong>If dynamicAllocation=true</strong> - how may it be possible that low consuming  memory spark job takes control of all Yarn resources </li>
  </ol>
  
  <p>Thanks</p>