Naive multiprocessing in Python with NumPy
Question: Despite the warnings and confused feelings I got from the ton of questions
that have been asked on the subject, especially on StackOverflow, I
paralellized a naive version of an embarassingly parallel problem (basically
**read-image-do-stuff-return** for a list of many images), returned the
resulting NumPy array for each computation and updated a global NumPy array
via the `callback` parameter, and immediately got a **x5 speedup** on a 8-core
machine.

Now, I probably didn't get x8 because of the lock required by each callback
call, but what I got is encouraging.

I'm trying to find out if this can be improved upon, or if this is a good
result. Questions :

  * I suppose the returned NumPy arrays got pickled?
  * Were the underlying NumPy buffers _copied_ or just _passed by reference_?
  * How can I find out what the bottleneck is? Any particularly useful technique?
  * Can I improve on that or is such an improvement pretty common in such cases?


Answer: I've had great success sharing large NumPy arrays (by reference, of course)
between multiple processes using `sharedmem` module:
<https://bitbucket.org/cleemesser/numpy-sharedmem>. Basically it suppresses
pickling that normally happens when passing around NumPy arrays. All you have
to do is, instead of:

    
    
    import numpy as np
    foo = np.empty(1000000)
    

do this:

    
    
    import sharedmem
    foo = sharedmem.empty(1000000)
    

and off you go passing `foo` from one process to another, like:

    
    
    q = multiprocessing.Queue()
    ...
    q.put(foo)
    

Note however, that this module has a known possibility of a memory leak upon
ungraceful program exit, described to some extent here:
<http://grokbase.com/t/python/python-list/1144s75ps4/multiprocessing-shared-
memory-vs-pickled-copies>.

Hope this helps. I use the module to speed up live image processing on multi-
core machines (my project is <https://github.com/vmlaker/sherlock>.)

