title: Python's multiprocessing: speed up a for-loop for several sets of parameters, "apply" vs. "apply_async"
body: <p>I would like to integrate a system of differential equations using a lot of different parameter combinations and store the variables’ final values that belong to a certain set of parameters. Therefore, I implemented a simple for-loop in which random initial conditions and parameter combinations are created, the system is integrated and the values of interest are stored in the respective arrays.
  Since I intend to do this for many parameter combinations for a rather complex system (here I only use a toy system for illustration), which can also become stiff, I would like to parallelize the simulations to speed up the process using Python’s “multiprocessing” module.</p>
  
  <p>However, when I run the simulations, the for-loop is always faster than its parallelized version. The only way to be faster than the for-loop I’ve found so far, is to use “apply_async” instead of “apply”. For 10 different parameter combinations, I get for example the following output (using the code from below):</p>
  
  <pre><code>The for loop took  0.11986207962 seconds!
  [ 41.75971761  48.06034375  38.74134139  25.6022232   46.48436046
    46.34952734  50.9073202   48.26035086  50.05026187  41.79483135]
  Using apply took  0.180637836456 seconds!
  41.7597176061
  48.0603437545
  38.7413413879
  25.6022231983
  46.4843604574
  46.3495273394
  50.9073202011
  48.2603508573
  50.0502618731
  41.7948313502
  Using apply_async took  0.000414133071899 seconds!
  41.7597176061
  48.0603437545
  38.7413413879
  25.6022231983
  46.4843604574
  46.3495273394
  50.9073202011
  48.2603508573
  50.0502618731
  41.7948313502
  </code></pre>
  
  <p>Although in this example the order of the results are identical for “apply” and “apply_async”, this seems not to be true in general. So, I would like to use “apply_async” since it is much faster but in this case I don’t know how I can match the outcome of the simulations to the parameters/initial conditions I used for the respective simulation.</p>
  
  <p>My questions are therefore:</p>
  
  <p>1) Why is “apply” much slowlier than the simple for-loop in this case?</p>
  
  <p>2) When I use “apply_async” instead of “apply“, the parallelized version becomes very much faster than the for-loop but how can I then match the outcome of the simulations to the parameters I used in the respective simulation?</p>
  
  <p>3) In this case, the results of “apply” and “apply_async” have the same order. Why is that? Coincidence?</p>
  
  <p>My code can be found below:</p>
  
  <pre><code>from pylab import *
  import multiprocessing as mp
  from scipy.integrate import odeint
  import time
  
  #my system of differential equations
  def myODE (yn,tvec,allpara):
  
      (x, y, z) = yn
  
      a, b = allpara['para']
  
      dx  = -x + a*y + x*x*y
      dy = b - a*y - x*x*y
      dz = x*y
  
      return (dx, dy, dz) 
  
  #for reproducibility    
  seed(0) 
  
  #time settings for integration
  dt = 0.01
  tmax = 50
  tval = arange(0,tmax,dt)
  
  numVar = 3 #number of variables (x, y, z)
  numPar = 2 #number of parameters (a, b)
  numComb = 10 #number of parameter combinations
  
  INIT = zeros((numComb,numVar)) #initial conditions will be stored here
  PARA = zeros((numComb,numPar)) #parameter combinations for a and b will be stored here
  RES = zeros(numComb) #z(tmax) will be stored here
  
  tic = time.time()
  
  for combi in range(numComb):
  
      INIT[combi,:] = append(10*rand(2),0) #initial conditions for x and y are randomly chosen, z is 0
  
      PARA[combi,:] = 10*rand(2) #parameter a and b are chosen randomly
  
      allpara = {'para': PARA[combi,:]}
  
      results = transpose(odeint(myODE, INIT[combi,:], tval, args=(allpara,))) #integrate system
  
      RES[combi] = results[numVar - 1][-1] #store z
  
      #INIT[combi,:] = results[:,-1] #update initial conditions
      #INIT[combi,-1] = 0 #set z to 0
  
  toc = time.time()
  
  print 'The for loop took ', toc-tic, 'seconds!'
  
  print RES
  
  #function for the multi-processing part
  def runMyODE(yn,tvec,allpara):
  
      return transpose(odeint(myODE, yn, tvec, args=(allpara,)))
  
  tic = time.time()
  
  pool = mp.Pool(processes=4)
  results = [pool.apply(runMyODE, args=(INIT[combi,:],tval,{'para': PARA[combi,:]})) for combi in range(numComb)]
  
  toc = time.time()
  
  print 'Using apply took ', toc-tic, 'seconds!'
  
  for sol in range(numComb):
      print results[sol][2,-1] #print final value of z
  
  tic = time.time()    
  resultsAsync = [pool.apply_async(runMyODE, args=(INIT[combi,:],tval,{'para': PARA[combi,:]})) for combi in range(numComb)]    
  toc = time.time()
  print 'Using apply_async took ', toc-tic, 'seconds!'
  
  for sol in range(numComb):
      print resultsAsync[sol].get()[2,-1] #print final value of z
  </code></pre>