title: cluster-mode SPARK refuses to run more than two jobs concurrently
body: <p>My Spark cluster refuses to run more than two jobs simultaneously. One of the three will invariable stay stuck in 'ACCEPTED' state.</p>
  
  <h1>Hardware</h1>
  
  <pre><code>4 Data Node with spark clients, 24gb ram, 4processors
  </code></pre>
  
  <h1>Cluster Metrics show there should be enough cores</h1>
  
  <pre><code>Apps Submitted    3
  Apps Pending    1
  Apps Running    2
  Apps Completed    0
  Containers Running   4
  Memory Used    8GB
  Memory Total  32GB
  Memory Reserved  0B
  VCores Used    4
  VCores Total    8
  VCores Reserved    0
  Active Nodes    2
  Decommissioned Nodes    0
  Lost Nodes    0
  Unhealthy Nodes   0
  Rebooted Nodes    0
  </code></pre>
  
  <h1>On Application Manager you can see the final the only way to run the third app is to kill a running one</h1>
  
  <pre><code>application_1504018580976_0002 adm com.x.app1 SPARK default 0 [date] N/A RUNNING UNDEFINED 2 2 5120 25.0 25.0 
  application_1500031233020_0090 adm com.x.app2 SPARK default 0 [date] N/A RUNNING UNDEFINED 2 2 3072 25.0 25.0 
  application_1504024737012_0001 adm com.x.app3 SPARK default 0 [date] N/A ACCEPTED UNDEFINED 0 0 0 0.0 0.0
  </code></pre>
  
  <p>The running apps have 2x containers and 2x allocated vcores, 25% of the queue and 25% of the cluster.</p>
  
  <h1>Deployment command for all 3 apps.</h1>
  
  <pre><code>/usr/hdp/current/spark2-client/bin/spark-submit 
  --master yarn 
  --deploy-mode cluster 
  --driver-cores 1 
  --driver-memory 512m 
  --num-executors 1 
  --executor-cores 1 
  --executor-memory 1G 
  --class com..x.appx ../lib/foo.jar
  </code></pre>
  
  <h1>Capacity Scheduler</h1>
  
  <pre><code>yarn.scheduler.capacity.default.minimum-user-limit-percent = 100
  yarn.scheduler.capacity.maximum-am-resource-percent = 0.2
  yarn.scheduler.capacity.maximum-applications = 10000
  yarn.scheduler.capacity.node-locality-delay = 40
  yarn.scheduler.capacity.root.accessible-node-labels = *
  yarn.scheduler.capacity.root.acl_administer_queue = *
  yarn.scheduler.capacity.root.capacity = 100
  yarn.scheduler.capacity.root.default.acl_administer_jobs = *
  yarn.scheduler.capacity.root.default.acl_submit_applications = *
  yarn.scheduler.capacity.root.default.capacity = 100
  yarn.scheduler.capacity.root.default.maximum-capacity = 100
  yarn.scheduler.capacity.root.default.state = RUNNING
  yarn.scheduler.capacity.root.default.user-limit-factor = 1
  yarn.scheduler.capacity.root.queues = default
  </code></pre>