title: Why does vcore always equal the number of nodes in Spark on YARN?
body: <p>I have a Hadoop cluster with 5 nodes, each of which has 12 cores with 32GB memory. I use YARN as MapReduce framework, so I have the following settings with YARN:</p>
  
  <ul>
  <li>yarn.nodemanager.resource.cpu-vcores=10</li>
  <li>yarn.nodemanager.resource.memory-mb=26100</li>
  </ul>
  
  <p>Then the cluster metrics shown on my YARN cluster page (<a href="http://myhost:8088/cluster/apps">http://myhost:8088/cluster/apps</a>) displayed that <b>VCores Total</b> is <b>40</b>. This is pretty fine!</p>
  
  <p>Then I installed Spark on top of it and use spark-shell in yarn-client mode.</p>
  
  <p>I ran one Spark job with the following configuration:</p>
  
  <ul>
  <li>--driver-memory 20480m </li>
  <li>--executor-memory 20000m </li>
  <li>--num-executors 4 </li>
  <li>--executor-cores 10 </li>
  <li>--conf spark.yarn.am.cores=2 </li>
  <li>--conf spark.yarn.executor.memoryOverhead=5600</li>
  </ul>
  
  <p>I set <b>--executor-cores</b> as <b>10</b>, <b>--num-executors</b> as <b>4</b>, so logically, there should be totally <b>40 Vcores Used</b>. However, when I check the same YARN cluster page after the Spark job started running, there are only  <b>4 Vcores Used</b>, and  <b>4 Vcores Total</b></p>
  
  <p><a href="https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/conf/capacity-scheduler.xml">I also found</a> that there is a parameter in <code>capacity-scheduler.xml</code> - called <code>yarn.scheduler.capacity.resource-calculator</code>:</p>
  
  <blockquote>
    <p>"The ResourceCalculator implementation to be used to compare Resources in the scheduler. The default i.e. DefaultResourceCalculator only uses Memory while DominantResourceCalculator uses dominant-resource to compare multi-dimensional resources such as Memory, CPU etc."</p>
  </blockquote>
  
  <p>I then changed that value to <code>DominantResourceCalculator</code>.</p>
  
  <p>But then when I restarted YARN and run the same Spark application, I still got the same result, say the cluster metrics still told that VCores used is 4! I also checked the CPU and memory usage on each node with htop command, I found that none of the nodes had all 10 CPU cores fully occupied. What can be the reason? </p>
  
  <p>I tried also to run the same Spark job in fine-grained way, say with <code>--num executors 40 --executor-cores 1</code>, in this ways I checked again the CPU status on each worker node, and all CPU cores are fully occupied.</p>