title: Why does Spark submit script spark-submit ignore `--num-executors`?
body: <p>We have Spark 1.0.0 running under YARN, and <code>--num-executors</code> does not seem to increase the number of executors nor nodes that are used.  I say I want 8, but I usually end up with between 3 and 5.  There are no errors in the output, which is what I would expect if nodes where down and couldn't be spoken to.</p>
  
  <p>NOTE: If you are NOT running under YARN, then num-executors will be ignored, e.g. spark standalone mode.  See accepted answer for solution and comment.</p>
  
  <p>UPDATE: If I ask for X resources I want X resources, and if I can't have them, I want to be put in a queue or given an error message of some sort.  This is because my job will fail if I don't get X resources - I know just how much resources I need before my job falls over.  I don't want to implement some extra layer over my job to check how many executors &amp; nodes I'm going to be given so that it can gracefully kill off the job before it blows up on it's own accord. So the second part of the question is "1) is there a way to tell YARN/Spark to fail if I can't get the executors I want? 2) force stop YARN from putting more than 1 executor on the same node"</p>
  
  <p>(In 0.9.0 this was not a problem, N nodes meant N workers and jobs would just queue)</p>