Python multiprocessing is taking much longer than single processing
Question: I am performing some large computations on 3 different numpy 2D arrays
sequentially. The arrays are huge, 25000x25000 each. Each computation takes
significant time so I decided to run 3 of them in parallel on 3 CPU cores on
the server. I am following standard multiprocessing guideline and creating 2
processes and a worker function. Two computations are running through the 2
processes and the third one is running locally without separate process. I am
passing the huge arrays as arguments of the processes like :

    
    
    p1 = Process(target = Worker, args = (queue1, array1, ...)) # Some other params also going
    
    p2 = Process(target = Worker, args = (queue2, array2, ...)) # Some other params also going
    

the Worker function sends back two numpy vectors (1D array) in a list appended
in the queue like:

    
    
    queue.put([v1, v2])
    

I am not using `multiprocessing.pool`

but surprisingly I am not getting speedup, it is actually running 3 times
slower. Is passing large arrays taking time? I am unable to figure out what is
going on. Should I use shared memory objects instead of passing arrays?

I shall be thankful if anybody can help.

Thank you.


Answer: Here is an example using `np.memmap` and `Pool`. See that you can define the
number of processes and workers. In this case you don't have control over the
queue, which can be achieved using `multiprocessing.Queue`:

    
    
    from multiprocessing import Pool
    
    import numpy as np
    
    def mysum(array_file_name, col1, col2, shape):
        a = np.memmap(array_file_name, shape=shape, mode='r+')
        a[:, col1:col2] = np.random.random((shape[0], col2-col1))
        ans = a[:, col1:col2].sum()
        del a
        return ans
    
    if __name__ == '__main__':
        nop = 1000 # number_of_processes
        now = 3 # number of workers
        p = Pool(now)
        array_file_name = 'test.array'
        shape = (250000, 250000)
        a = np.memmap(array_file_name, shape=shape, mode='w+')
        del a
        cols = [[shape[1]*i/nop, shape[1]*(i+1)/nop] for i in range(nop)]
        results = []
        for c1, c2 in cols:
            r = p.apply_async(mysum, args=(array_file_name, c1, c2, shape))
            results.append(r)
        p.close()
        p.join()
    
        final_result = sum([r.get() for r in results])
        print final_result
    

You can achieve better performances using shared memory parallel processing,
when possible. See this related question:

  * [Shared-memory objects in python multiprocessing](http://stackoverflow.com/q/10721915/832621)

