title: Unable to create Scheduler Pools in EMR using PySpark
body: <p>I am fairly new to the concept of Spark Schedulers/Pooling and need to implement the same in one of my Projects. Just in order to understand the concept better, I scribbled the following streaming PySpark Code on my local and executed :</p>
  <pre><code>from pyspark.sql import SparkSession
  import threading
  
  
  def do_job(f1, f2):
      df1 = spark.readStream.json(f1)
      df2 = spark.readStream.json(f2)
      df = df1.join(df2, &quot;id&quot;, &quot;inner&quot;)
      df.writeStream.format(&quot;parquet&quot;).outputMode(&quot;append&quot;) \
          .option(&quot;checkpointLocation&quot;, &quot;checkpoint&quot; + str(f1) + &quot;/&quot;) \
          .option(&quot;path&quot;, &quot;Data/Sample_Delta_Data/date=A&quot; + str(f1)) \
          .start()
  
      # outputs.append(df1.join(df2, &quot;id&quot;, &quot;inner&quot;).count())
  
  
  if __name__ == &quot;__main__&quot;:
      spark = SparkSession \
          .builder \
          .appName(&quot;Demo&quot;) \
          .master(&quot;local[4]&quot;) \
          .config(&quot;spark.sql.autoBroadcastJoinThreshold&quot;, &quot;50B&quot;) \
          .config(&quot;spark.scheduler.mode&quot;, &quot;FAIR&quot;) \
          .config(&quot;spark.sql.streaming.schemaInference&quot;, &quot;true&quot;) \
          .getOrCreate()
  
      file_prefix = &quot;data_new/data/d&quot;
      jobs = []
      outputs = []
  
      for i in range(0, 6):
          file1 = file_prefix + str(i + 1)
          file2 = file_prefix + str(i + 2)
          thread = threading.Thread(target=do_job, args=(file1, file2))
          jobs.append(thread)
  
      for j in jobs:
          j.start()
  
      for j in jobs:
          j.join()
  
      spark.streams.awaitAnyTermination()
      # print(outputs)
  </code></pre>
  <p>As could be seen above, I am using FAIR Scheduler option and using 'Threading Library' in PySpark  to implement Pooling.</p>
  <p>As the matter fact, the above code is creating <strong>pools</strong> on my Local System but when I run the same on AWS EMR cluster, no Pools are getting created.</p>
  <p>Am I missing something specific to AWS EMR ?</p>
  <p>Suggestions please!</p>
  <p>Regards.</p>