title: Force YARN to deploy Spark tasks across all slaves
body: <p>I've been deploying the SparkPi example on an 8-node cluster. It would appear that the tasks associated with the example are not being deployed to all nodes in the cluster, even though the cluster is under-utilized (no other jobs running).</p>
  
  <p>Here's how I am starting the SparkPi example:</p>
  
  <pre><code>spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --driver-memory 1g --executor-memory 1g --executor-cores 1 --num-executors 7 $SPARK_HOME/lib/spark-examples-1.6.1-hadoop2.6.0.jar 100000
  </code></pre>
  
  <p>However, when I look at which nodes are being utilized, this is what I see:
  <a href="https://i.stack.imgur.com/UqVn8.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/UqVn8.png" alt="hadoop cluster"></a></p>
  
  <p>I have a feeling this is because I'm using the <code>CapacityScheduler</code> in the resource manager. Here is my <code>yarn-site.xml</code> file:</p>
  
  <pre><code>&lt;configuration&gt;
      &lt;property&gt;
          &lt;name&gt;yarn.nodemanager.aux-services&lt;/name&gt;
          &lt;value&gt;mapreduce_shuffle&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
        &lt;name&gt;yarn.resourcemanager.hostname&lt;/name&gt;
        &lt;value&gt;master.cluster&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
        &lt;name&gt;yarn.resourcemanager.address&lt;/name&gt;
        &lt;value&gt;master.cluster:8032&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
        &lt;name&gt;yarn.resourcemanager.scheduler.address&lt;/name&gt;
        &lt;value&gt;master.cluster:8030&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
        &lt;name&gt;yarn.resourcemanager.webapp.address&lt;/name&gt;
        &lt;value&gt;0.0.0.0:8088&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
        &lt;name&gt;yarn.resourcemanager.resource-tracker.address&lt;/name&gt;
        &lt;value&gt;master.cluster:8031&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
        &lt;name&gt;yarn.resourcemanager.admin.address&lt;/name&gt;
        &lt;value&gt;master.cluster:8033&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
        &lt;name&gt;yarn.resourcemanager.admin.address&lt;/name&gt;
        &lt;value&gt;master.cluster:8033&lt;/value&gt;
      &lt;/property&gt;
  
      &lt;property&gt;
        &lt;name&gt;yarn.application.classpath&lt;/name&gt;
        &lt;value&gt;/usr/local/hadoop/etc/hadoop, /usr/local/hadoop/share/hadoop/common/*, /usr/local/hadoop/share/hadoop/common/lib/*, /usr/local/hadoop/share/hadoop/hdfs/*, /usr/local/hadoop/share/hadoop/hdfs/lib/*, /usr/local/hadoop/share/hadoop/mapreduce/*, /usr/local/hadoop/share/hadoop/mapreduce/lib/*, /usr/local/hadoop/share/hadoop/yarn/*, /usr/local/hadoop/share/hadoop/yarn/lib/*&lt;/value&gt;
      &lt;/property&gt;
  
      &lt;property&gt;
      &lt;description&gt;
        Number of seconds after an application finishes before the nodemanager's
        DeletionService will delete the application's localized file directory
        and log directory.
  
        To diagnose Yarn application problems, set this property's value large
        enough (for example, to 600 = 10 minutes) to permit examination of these
        directories. After changing the property's value, you must restart the
        nodemanager in order for it to have an effect.
  
        The roots of Yarn applications' work directories is configurable with
        the yarn.nodemanager.local-dirs property (see below), and the roots
        of the Yarn applications' log directories is configurable with the
        yarn.nodemanager.log-dirs property (see also below).
      &lt;/description&gt;
      &lt;name&gt;yarn.nodemanager.delete.debug-delay-sec&lt;/name&gt;
      &lt;value&gt;600&lt;/value&gt;
    &lt;/property&gt;
  
  &lt;/configuration&gt;
  </code></pre>
  
  <p>How can I tweak YARN so that it deploys tasks across <em>all</em> nodes?</p>