title: Spark jobs seem to only be using a small amount of resources
body: <p>Please bear with me because I am still quite new to Spark.</p>
  
  <p>I have a GCP DataProc cluster which I am using to run a large number of Spark jobs, 5 at a time.</p>
  
  <p>Cluster is 1 + 16, 8 cores / 40gb mem / 1TB storage per node.</p>
  
  <p>Now I might be misunderstanding something or not doing something correctly, but I currently have 5 jobs running at once, and the Spark UI is showing that only 34/128 vcores are in use, and they do not appear to be evenly distributed (The jobs were executed simultaneously, but the distribution is 2/7/7/11/7. There is only one core allocated per running container.</p>
  
  <p>I have used the flags <code>--executor-cores 4</code> and <code>--num-executors 6</code> which doesn't seem to have made any difference.</p>
  
  <p>Can anyone offer some insight/resources as to how I can fine tune these jobs to use all available resources?</p>