title: AWS EMR cluster - spark-submit Should you be able to run more than one job at a time?
body: <p>I have a Python script that process all the data for a U.S. state. I pass the two letter state code in as the first argument to the script.</p>
  <p>So in an ssh session connected to the master node in AWS EMR I can submit a long job like:</p>
  <pre><code>spark-submit --master yarn --deploy-mode client --num-executors 1 ./process_state.py VT
  </code></pre>
  <p>If I go into another ssh to master and run</p>
  <pre><code>spark-submit --master yarn --deploy-mode client --num-executors 1 ./process_state.py WY
  </code></pre>
  <p>Internally the state code gets concatenated onto the application name. I thought that might be the issue at first. But the second submit-spark will just sit there and do nothing. With the message:</p>
  <pre><code>21/05/04 16:42:46 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  </code></pre>
  <p>I'm not sure what to change or do to alter the behavior and be able to process more at one time.
  On AWS EMR I have 1 master, 2 cores, and 2 task machines. Shouldn't think I'd be out of resources from one script. I'm new to Spark/PySpark so it may be something simple.</p>