title: Performance issues for spark on YARN
body: <p>We are trying to run our spark cluster on yarn. We are having some performance issues especially when compared to the standalone mode. </p>
  
  <p>We have a cluster of 5 nodes with each having 16GB RAM and 8 cores each. We have configured the minimum container size as 3GB and maximum as 14GB in yarn-site.xml. When submitting the job to yarn-cluster we supply number of executor = 10, memory of executor =14 GB. According to my understanding our job should be allocated 4 container of 14GB. But the spark UI shows only 3 container of 7.2GB each. </p>
  
  <p>We are unable to ensure the container number and resources allocated to it. This causes detrimental performance when compared to the standalone mode. </p>
  
  <p>Can you drop any pointer on how to optimize yarn performance?</p>
  
  <p>This is the command I use for submitting the job: </p>
  
  <pre><code>$SPARK_HOME/bin/spark-submit --class "MyApp" --master yarn-cluster --num-executors 10 --executor-memory 14g  target/scala-2.10/my-application_2.10-1.0.jar  
  </code></pre>
  
  <p>Following the discussion I changed my yarn-site.xml file and also the spark-submit command.</p>
  
  <p>Here is the new yarn-site.xml code :</p>
  
  <pre><code>&lt;property&gt;
  &lt;name&gt;yarn.resourcemanager.hostname&lt;/name&gt;
  &lt;value&gt;hm41&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.nodemanager.resource.memory-mb&lt;/name&gt;
  &lt;value&gt;14336&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.scheduler.minimum-allocation-mb&lt;/name&gt;
  &lt;value&gt;2560&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.scheduler.maximum-allocation-mb&lt;/name&gt;
  &lt;value&gt;13312&lt;/value&gt;
  &lt;/property&gt;
  </code></pre>
  
  <p>And the new command for spark submit is</p>
  
  <pre><code>$SPARK_HOME/bin/spark-submit --class "MyApp" --master yarn-cluster --num-executors 4 --executor-memory  10g --executor-cores 6   target/scala-2.10/my-application_2.10-1.0.jar 
  </code></pre>
  
  <p>With this I am able to get 6 cores on each machine but the memory usage of each node is still around 5G. I have attached the screen shot of SPARKUI and htop.
  <img src="https://i.stack.imgur.com/bqZwU.png" alt="enter image description here">
  <img src="https://i.stack.imgur.com/CQM1a.png" alt="Spark UI Screenshot![][1]"></p>