title: How to run multiple on demand jobs on same spark context
body: <p>I want to run different jobs on demand with same spark context, but i don't know how exactly i can do this.</p>
  
  <p>I try to get current context, but seems it create a new spark context(with new executors).</p>
  
  <p>I call spark-submit to add new jobs.</p>
  
  <p>I run code on Amazon EMR, with yarn as resource manager.</p>
  
  <p>My code:</p>
  
  <pre><code>val sparkContext = SparkContext.getOrCreate()
  val content = 1 to 40000
  val result = sparkContext.parallelize(content, 5)
  result.map(value =&gt; value.toString).foreach(loop)
  
  def loop(x: String): Unit = {
     for (a &lt;- 1 to 30000000) {
  
     }
  }
  </code></pre>
  
  <p>spark-submit:</p>
  
  <pre><code>spark-submit --executor-cores 1 \
               --executor-memory 1g \
               --driver-memory 1g \
               --master yarn \
               --deploy-mode cluster \
               --conf spark.dynamicAllocation.enabled=true \
               --conf spark.shuffle.service.enabled=true \
               --conf spark.dynamicAllocation.minExecutors=1 \
               --conf spark.dynamicAllocation.maxExecutors=3 \
               --conf spark.dynamicAllocation.initialExecutors=3 \
               --conf spark.executor.instances=3 \
  </code></pre>
  
  <p>If i run twice spark-submit it create 6 executors, but i want to run all this jobs on same spark application.</p>
  
  <p>How can achieve adding jobs to an existing spark application?</p>
  
  <p>I read about JobServer(<a href="https://github.com/spark-jobserver/spark-jobserver" rel="nofollow noreferrer">https://github.com/spark-jobserver/spark-jobserver</a>) which achieved what i want to do, but i don't understand how they do this.</p>