title: Python multiprocessing: How to know to use Pool or Process?
body: <p>So I have an algorithm I am writing, and the function <code>multiprocess</code> is supposed to call another function, <code>CreateMatrixMp()</code>, on as many processes as there are cpus, in parallel. I have never done multiprocessing before, and cannot be certain which one of the below methods is more efficient. The word "efficient" being used in the context of the function <code>CreateMatrixMp()</code> needing to potentially be called thousands of times.I have read all of the documentation on the python <code>multiprocessing</code> module, and have come to these two possibilities:</p>
  
  <p>First is using the <code>Pool</code> class:</p>
  
  <pre><code>def MatrixHelper(self, args):
      return self.CreateMatrix(*args)
  
  def Multiprocess(self, sigmaI, sigmaX):
  
      cpus = mp.cpu_count()
      print('Number of cpu\'s to process WM: %d' % cpus)
      poolCount = cpus*2
      args = [(sigmaI, sigmaX, i) for i in range(self.numPixels)]
  
      pool = mp.Pool(processes = poolCount, maxtasksperchild= 2)
      tempData = pool.map(self.MatrixHelper, args)
      pool.close()
      pool.join()
  </code></pre>
  
  <p>And next is using the <code>Process</code> class:</p>
  
  <pre><code>def Multiprocess(self, sigmaI, sigmaX):
  
      cpus = mp.cpu_count()
      print('Number of cpu\'s to process WM: %d' % cpus)
  
      processes = [mp.Process(target = self.CreateMatrixMp, args = (sigmaI, sigmaX, i,)) for i in range(self.numPixels)]
      for p in processes:
          p.start()
      for p in processes:
          p.join()
  </code></pre>
  
  <p><code>Pool</code> seems to be the better choice. I have read that it causes less overhead. And <code>Process</code> does not consider the number of cpus on the machine. The only problem is that using <code>Pool</code> in this manner gives me error after error, and whenever I fix one, there is a new one underneath it. <code>Process</code> seems easier to implement, and for all I know it may be the better choice. What does your experience tell you?</p>
  
  <p>If <code>Pool</code> should be used, then am I right in choosing <code>map()</code>? It would be preferred that order is maintained. I have <code>tempData = pool.map(...)</code> because the <code>map</code> function is supposed to return a list of the results of every process. I am not sure how <code>Process</code> handles its returned data. </p>