title: Spark on Yarn not using all nodes
body: <p>I have a 9 node Hadoop cluster, running MapR with the following Yarn roles:</p>
  
  <ul>
  <li>1 node is a Yarn ResourceManager</li>
  <li>8 of the nodes are Yarn NodeManagers (two of which are also backup ResourceManagers)</li>
  </ul>
  
  <p>Whenever I submit a Spark job via pyspark I see something like the following:</p>
  
  <pre><code>yarn node -list
  17/04/26 23:40:26 INFO client.MapRZKBasedRMFailoverProxyProvider: Updated RM address to hadoop-n1/XXX.XXX.XX.XX:8032
  Total Nodes:8
           Node-Id             Node-State Node-Http-Address       Number-of-Running-Containers
   hadoop-d8:52118                RUNNING    hadoop-d8:8042                                  4
   hadoop-d5:39108                RUNNING    hadoop-d5:8042                                  0
   hadoop-d2:42471                RUNNING    hadoop-d2:8042                                  0
   hadoop-d4:36191                RUNNING    hadoop-d4:8042                                  0
   hadoop-d3:53476                RUNNING    hadoop-d3:8042                                  0
   hadoop-d6:52497                RUNNING    hadoop-d6:8042                                  4
   hadoop-d1:59887                RUNNING    hadoop-d1:8042                                  0
   hadoop-d7:51878                RUNNING    hadoop-d7:8042                                  4
  </code></pre>
  
  <p>So <code>d6</code>, <code>d7</code> and <code>d8</code> are always used (and always with 4 containers) and the other nodes are never used. I've set <code>spark.dynamicAllocation.enabled</code> to <code>True</code> which made no difference. I've tried restarting all Yarn processes, again with no luck. <strong>How can I get my jobs to run on all nodes?</strong></p>
  
  <p>I am running Hadoop version 2.7.0, Spark version 2.0.1 and MapR version 5.2 on Ubuntu 14.04. The Yarn resource calculator is set to <code>DiskBasedResourceCalculator</code> which uses uses Memory, CPU and Disk. </p>
  
  <p><strong>EDIT:</strong> Based on <a href="https://stackoverflow.com/questions/33940884/why-does-yarn-on-emr-not-allocate-all-nodes-to-running-spark-jobs?rq=1">this</a> answer suggested in the comments I changed to the <code>DominantResourceCalculator</code> and set <code>spark.executor.instances</code> to 0. Still no changes. If I look at one of the unused nodes I see:</p>
  
  <pre><code>% yarn node -status hadoop-d7:35731
  17/04/27 16:01:19 INFO client.MapRZKBasedRMFailoverProxyProvider: Updated RM address to hadoop-n1/    Node Report : 
      Node-Id : hadoop-d7:35731
      Rack : /default-rack
      Node-State : RUNNING
      Node-Http-Address : hadoop-d7:8042
      Last-Health-Update : Thu 27/Apr/17 03:59:29:475EDT
      Health-Report : 
      Containers : 4
      Memory-Used : 32768MB
      Memory-Capacity : 97988MB
      CPU-Used : 20 vcores
      CPU-Capacity : 20 vcores
      Node-Labels : 
  </code></pre>
  
  <p>whereas the good nodes show</p>
  
  <pre><code>% yarn node -status hadoop-d2:37314
  17/04/27 16:01:05 INFO client.MapRZKBasedRMFailoverProxyProvider: Updated RM address to hadoop-n1/
  Node Report : 
      Node-Id : hadoop-d2:37314
      Rack : /default-rack
      Node-State : RUNNING
      Node-Http-Address : hadoop-d2:8042
      Last-Health-Update : Thu 27/Apr/17 03:59:32:255EDT
      Health-Report : 
      Containers : 0
      Memory-Used : 0MB
      Memory-Capacity : 70189MB
      CPU-Used : 0 vcores
      CPU-Capacity : 4 vcores
      Node-Labels : 
  </code></pre>