title: scipy parallel cdist with multiprocessing
body: <p>I have a big matrix with millions of rows and hundreds of columns.
  The first <code>n</code> rows (about 100K) are reference rows, and for the others, I would like to find the <code>k</code> (about 10) closest neighbours in the reference vectors with <a href="http://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.cdist.html#scipy.spatial.distance.cdist" rel="nofollow">scipy cdist</a></p>
  
  <p>I created an <code>multiprocessing.sharedctypes.Array</code> from the matrix, and use <code>asarray</code> and slicing to split up the matrix and compute distances with <code>cdist</code>.</p>
  
  <p>My current code looks like this:</p>
  
  <pre class="lang-py prettyprint-override"><code>import numpy
  from multiprocessing import Pool, sharedctypes
  from scipy.spatial.distance import cdist
  
  shared_m = None
  
  def generate_sample():
      m = numpy.random.random((20000, 640))
      shape = m.shape
      global shared_m
      shared_m = sharedctypes.Array('d', m.flatten(), lock=False)
      return shape
  
  def dist(A, B, metric):
      return cdist(A, B, metric=metric)
  
  def get_closest(args):
      shape, lenA, start_B, end_B, metric, numres = args
      m = numpy.ctypeslib.as_array(shared_m)
      m.shape = shape
      A = numpy.asarray(m[:lenA,:], dtype=numpy.double)
      B = numpy.asarray(m[start_B:end_B,:], dtype=numpy.double)
      distances = dist(B, A, metric)
      # rest of code to find closests
  
  def p_get_closest(shape, lenA, lenB, metric="cosine", sample_size=1000, numres=10):
      p = Pool(4)
      args = ((shape, lenA, i, i + sample_size, metric, numres)
              for i in xrange(lenB / sample_size))
      res = p.map_async(get_closest, args)
      return res.get()
  
  def main():
      shape = generate_sample()
      p_get_closest(shape, 5000, shape[0] - 5000, "cosine", 3000, 10)
  
  if __name__ == "__main__":
      main()
  </code></pre>
  
  <p>My problem right now is that the parallel calls of cdist are somehow block each other. (Maybe I use the block expression incorrectly. The problem is that there are no parallel cdist computations)</p>
  
  <p>I tried to trace back the problem with printouts into <code>scipy/spatial/distance.py</code> and <code>scipy/spatial/src/distance.c</code> to understand where the run blocks. It looks like there is no copying of data, the <code>dtypes</code> argument took care of that.</p>
  
  <p>When putting printf into <code>distance.c:cdist_cosine()</code>, it shows that all the processes reach the point where the actual computation starts (before the for loops), but the computations don't run in parallel.</p>
  
  <p>I tried a lot of things like using <code>multiprocessing.sharedctypes.RawArray</code> instead of <code>Array</code>, using <code>lock=True</code> while creating the <code>Array</code>.</p>
  
  <p>I have no other idea what I did wrong or how to investigate more the problem.</p>