title: Why does Yarn on EMR not allocate all nodes to running Spark jobs?
body: <p>I'm running a job on Apache Spark on Amazon Elastic Map Reduce (EMR).  Currently I'm running on emr-4.1.0 which includes Amazon Hadoop 2.6.0 and Spark 1.5.0.</p>
  
  <p>When I start the job, YARN correctly has allocated all the worker nodes to the spark job (with one for the driver, of course).</p>
  
  <p>I have the magic "maximizeResourceAllocation" property set to "true", and the spark property "spark.dynamicAllocation.enabled" also set to "true".</p>
  
  <p>However, if I resize the emr cluster by adding nodes to the CORE pool of worker machines, YARN only adds <em>some</em> of the new nodes to the spark job.</p>
  
  <p>For example, this morning I had a job that was using 26 nodes (m3.2xlarge, if that matters) - 1 for the driver, 25 executors.  I wanted to speed up the job so I tried adding 8 more nodes.  YARN has picked up all of the new nodes, but only allocated 1 of them to the Spark job.  Spark did successfully pick up the new node and is using it as an executor, but my question is why is YARN letting the other 7 nodes just sit idle?</p>
  
  <p>It's annoying for obvious reasons - I have to pay for the resources even though they're not being used, and my job hasn't sped up at all!</p>
  
  <p>Anybody know how YARN decides when to add nodes to running spark jobs?  What variables come into play?  Memory?  V-Cores?  Anything?</p>
  
  <p>Thanks in advance!</p>