title: distributing tasks evenly across executors
body: <p>Trying to read in csv files (<code>spark.read.csv("s3://...)</code>) stored in S3 using EMR. There are approximately 100 files in the S3 bucket, and ~300 instances in the EMR cluster each with two executors (7 cores each). I can see in the spark UI that nearly all tasks are allocated to just 10 executors, each with 10 tasks, rather than each task being assigned to different executors to take advantage of processing each file in parallel. </p>
  
  <p>Is there a knob to force the tasks to distribute across all executors?</p>
  
  <p><code>spark.dynamicAllocation.enabled</code> is set to  <code>false</code>, <code>spark.executor.instances</code> is set to the number of executors in the cluster.</p>
  
  <p><a href="https://i.stack.imgur.com/JPKvh.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/JPKvh.png" alt="enter image description here"></a></p>