title: Why do identical jobs run at different speeds with python's multiprocessing.Pool?
body: <p>I am trying to speed up some heavy simulations by using python's multiprocessing module on a machine with 24 cores that runs Suse Linux. From reading through the documentation, I understand that this only makes sense if the individual calculations take much longer than the overhead for creating the pool etc. </p>
  
  <p>What confuses me is that the execution time of some of the individual processes is much longer with multiprocessing than when I just run a single process. In my actual simulations the the time increases from around 300s to up to 1500s. Interestingly this gets worse when I use more processes. </p>
  
  <p>The following example illustrates the problem with a slightly shorter dummy loop:</p>
  
  <pre class="lang-py prettyprint-override"><code>from time import clock,time
  import multiprocessing
  import os
  
  
  def simulate(params):
      t1 = clock()
      result = 0
      for i in range(10000):
          for j in range(10000):
              result+=i*j
      pid = os.getpid()
      print 'pid: ',pid,' sim time: ',clock() - t1, 'seconds'
      return result
  
  if __name__ == '__main__':
  
  
      for n_procs in [1,5,10,20]:
          print n_procs,' processes:'
          t1 = time()
          result = multiprocessing.Pool(processes = n_procs).map(simulate,range(20))
          print 'total: ',time()-t1
  </code></pre>
  
  <p>This produces the following output:</p>
  
  <pre><code>1  processes:
  pid:  1872  sim time:  8.1 seconds
  pid:  1872  sim time:  7.92 seconds
  pid:  1872  sim time:  7.93 seconds
  pid:  1872  sim time:  7.89 seconds
  pid:  1872  sim time:  7.87 seconds
  pid:  1872  sim time:  7.74 seconds
  pid:  1872  sim time:  7.83 seconds
  pid:  1872  sim time:  7.84 seconds
  pid:  1872  sim time:  7.88 seconds
  pid:  1872  sim time:  7.82 seconds
  pid:  1872  sim time:  8.83 seconds
  pid:  1872  sim time:  7.91 seconds
  pid:  1872  sim time:  7.97 seconds
  pid:  1872  sim time:  7.84 seconds
  pid:  1872  sim time:  7.87 seconds
  pid:  1872  sim time:  7.91 seconds
  pid:  1872  sim time:  7.86 seconds
  pid:  1872  sim time:  7.9 seconds
  pid:  1872  sim time:  7.96 seconds
  pid:  1872  sim time:  7.97 seconds
  total:  159.337743998
  5  processes:
  pid:  1906  sim time:  8.66 seconds
  pid:  1907  sim time:  8.74 seconds
  pid:  1908  sim time:  8.75 seconds
  pid:  1905  sim time:  8.79 seconds
  pid:  1909  sim time:  9.52 seconds
  pid:  1906  sim time:  7.72 seconds
  pid:  1908  sim time:  7.74 seconds
  pid:  1907  sim time:  8.26 seconds
  pid:  1905  sim time:  8.45 seconds
  pid:  1909  sim time:  9.25 seconds
  pid:  1908  sim time:  7.48 seconds
  pid:  1906  sim time:  8.4 seconds
  pid:  1907  sim time:  8.23 seconds
  pid:  1905  sim time:  8.33 seconds
  pid:  1909  sim time:  8.15 seconds
  pid:  1908  sim time:  7.47 seconds
  pid:  1906  sim time:  8.19 seconds
  pid:  1907  sim time:  8.21 seconds
  pid:  1905  sim time:  8.27 seconds
  pid:  1909  sim time:  8.1 seconds
  total:  35.1368539333
  10  processes:
  pid:  1918  sim time:  8.79 seconds
  pid:  1920  sim time:  8.81 seconds
  pid:  1915  sim time:  14.78 seconds
  pid:  1916  sim time:  14.78 seconds
  pid:  1914  sim time:  14.81 seconds
  pid:  1922  sim time:  14.81 seconds
  pid:  1913  sim time:  14.98 seconds
  pid:  1921  sim time:  14.97 seconds
  pid:  1917  sim time:  15.13 seconds
  pid:  1919  sim time:  15.13 seconds
  pid:  1920  sim time:  8.26 seconds
  pid:  1918  sim time:  8.34 seconds
  pid:  1915  sim time:  9.03 seconds
  pid:  1921  sim time:  9.03 seconds
  pid:  1916  sim time:  9.39 seconds
  pid:  1913  sim time:  9.27 seconds
  pid:  1914  sim time:  12.12 seconds
  pid:  1922  sim time:  12.17 seconds
  pid:  1917  sim time:  12.15 seconds
  pid:  1919  sim time:  12.17 seconds
  total:  27.4067809582
  20  processes:
  pid:  1941  sim time:  8.63 seconds
  pid:  1939  sim time:  10.32 seconds
  pid:  1931  sim time:  12.35 seconds
  pid:  1936  sim time:  12.23 seconds
  pid:  1937  sim time:  12.82 seconds
  pid:  1942  sim time:  12.73 seconds
  pid:  1932  sim time:  13.01 seconds
  pid:  1946  sim time:  13.0 seconds
  pid:  1945  sim time:  13.74 seconds
  pid:  1944  sim time:  14.03 seconds
  pid:  1929  sim time:  14.44 seconds
  pid:  1943  sim time:  14.75 seconds
  pid:  1935  sim time:  14.8 seconds
  pid:  1930  sim time:  14.79 seconds
  pid:  1927  sim time:  14.85 seconds
  pid:  1934  sim time:  14.8 seconds
  pid:  1928  sim time:  14.83 seconds
  pid:  1940  sim time:  14.88 seconds
  pid:  1933  sim time:  15.05 seconds
  pid:  1938  sim time:  15.06 seconds
  total:  15.1311581135
  </code></pre>
  
  <p>What I do not understand is that <em>some</em> of the processes become much slower above a certain number of CPUs. I should add that nothing else is running on this machine. Is this expected? Am I doing something wrong?  </p>