title: Can Spark parallelize nested?
body: <p>I run the following code on a 60-code EC2 instance</p>
  
  <pre><code>from pyspark import SparkContext
  import time, md5
  workers_count = 10
  sc = SparkContext("local[%s]" % workers_count, "App Name")
  max_num = 50000000
  start_time = time.time()
  first_item = sc.parallelize(xrange(max_num)).map(lambda n: (n, md5.md5(str(n)).hexdigest())).reduce(lambda a,b: a if a[1] &gt; b[1] else b)
  end_time = time.time()
  print("sorting took took %s seconds with %s workers" % (end_time-start_time, workers_count))
  </code></pre>
  
  <p>with 1 worker it took 52 sec.
  with 2 workers it took 26 sec.
  with 4 workers it took 13 sec
  with 8 workers it took 6 sec
  with 16 or more workers it took 4 sec (more or less)</p>
  
  <p>The above code is the inner-part, and it needs to run a few million times</p>
  
  <p>From the above I understand that there is a limit to how much will parallelization improve the performance, which is OK, but since I'm using a 60-cores machine, and I want it to make the best usage of the cores, I want each loop to use 8 cores, having 7 loops running simultaneously.</p>
  
  <p>Is it possible to define for each function how many cores it will use?</p>