title: How to Distribute Multiprocessing Pool to Spark Workers
body: <p>I am trying to use multiprocessing to read 100 csv files in parallel (and subsequently process them separately in parallel). Here is my code running in Jupyter hosted on my EMR master node in AWS. (Eventually it will be 100k csv files hence the need for distributed reading).</p>
  <pre><code>import findspark
  import boto3
  from multiprocessing.pool import ThreadPool
  import logging
  import sys
  findspark.init()
  from pyspark import SparkContext, SparkConf, sql
  conf = SparkConf().setMaster(&quot;local[*]&quot;)
  conf.set('spark.scheduler.mode', 'FAIR')
  sc = SparkContext.getOrCreate(conf)
  spark = sql.SparkSession.builder.master(&quot;local[*]&quot;).appName(&quot;ETL&quot;).getOrCreate()
  s3 = boto3.resource(...)
  bucket = ''
  bucketObj = s3.Bucket(bucket)
  numNodes = 64
  def processTest(key):
      logger.info(key + ' ---- Start\n')
      fLog = spark.read.option(&quot;header&quot;, &quot;true&quot;) \
                           .option(&quot;inferSchema&quot;, &quot;true&quot;) \
                           .csv(buildS3Path(bucket) + key)
      logger.info(key + ' ---- Finish Read\n')
      fLog = renameColumns(NAME_MAP, fLog)
      logger.info(key + ' ---- Finish Rename\n')
      (landLog, flags) = validate(fLog)
      logger.info(key + ' ---- Finish Validation\n')
  
  files = list(bucketObj.objects.filter(Prefix=subfolder))
  keys = list(map(lambda obj: obj.key, files))
  keys = keys
  # files = s3C.list_objects(Bucket=bucket, Prefix=subfolder)['Contents']
  p = ThreadPool(numNodes)
  p.map(processTest, keys)
  </code></pre>
  <p>It runs fine except that it only uses the master node.
  <a href="https://i.stack.imgur.com/VNFO3.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/VNFO3.png" alt="The blue line is my master node." /></a></p>
  <p>The blue line is the CPU usage on my master node.
  All the logs show that I'm running on one machine:</p>
  <pre><code> INFO:pyspark:172.31.29.33
  </code></pre>
  <p>How do I make spark distribute the pool to the workers?</p>