Using multiprocessing.Manager.list instead of a real list makes the calculation take ages
Question: I wanted to try different ways of using `multiprocessing` starting with this
example:

    
    
    $ cat multi_bad.py 
    import multiprocessing as mp
    from time import sleep
    from random import randint
    
    def f(l, t):
    #   sleep(30)
        return sum(x < t for x in l)
    
    if __name__ == '__main__':
        l = [randint(1, 1000) for _ in range(25000)]
        t = [randint(1, 1000) for _ in range(4)]
    #   sleep(15)
        pool = mp.Pool(processes=4)
        result = pool.starmap_async(f, [(l, x) for x in t])
        print(result.get())
    

Here, `l` is a list that gets copied 4 times when 4 processes are spawned. To
avoid that, the documentation page offers using queues, shared arrays or proxy
objects created using `multiprocessing.Manager`. For the last one, I changed
the definition of `l`:

    
    
    $ diff multi_bad.py multi_good.py 
    10c10,11
    <     l = [randint(1, 1000) for _ in range(25000)]
    ---
    >     man = mp.Manager()
    >     l = man.list([randint(1, 1000) for _ in range(25000)])
    

The results still look correct, but the execution time has increased so
dramatically that I think I'm doing something wrong:

    
    
    $ time python multi_bad.py 
    [17867, 11103, 2021, 17918]
    
    real    0m0.247s
    user    0m0.183s
    sys 0m0.010s
    
    $ time python multi_good.py 
    [3609, 20277, 7799, 24262]
    
    real    0m15.108s
    user    0m28.092s
    sys 0m6.320s
    

The docs do say that this way is slower than shared arrays, but this just
feels wrong. I'm also not sure how I can profile this to get more information
on what's going on. Am I missing something?

P.S. With shared arrays I get times below 0.25s.

P.P.S. This is on Linux and Python 3.3.


Answer: Linux uses [copy-on-write](http://en.wikipedia.org/wiki/Copy-on-write) when
subprocesses are `os.fork`ed. To demonstrate:

    
    
    import multiprocessing as mp
    import numpy as np
    import logging
    import os
    
    logger = mp.log_to_stderr(logging.WARNING)
    
    def free_memory():
        total = 0
        with open('/proc/meminfo', 'r') as f:
            for line in f:
                line = line.strip()
                if any(line.startswith(field) for field in ('MemFree', 'Buffers', 'Cached')):
                    field, amount, unit = line.split()
                    amount = int(amount)
                    if unit != 'kB':
                        raise ValueError(
                            'Unknown unit {u!r} in /proc/meminfo'.format(u = unit))
                    total += amount
        return total
    
    def worker(i):
        x = data[i,:].sum()    # Exercise access to data
        logger.warn('Free memory: {m}'.format(m = free_memory()))
    
    def main():
        procs = [mp.Process(target = worker, args = (i, )) for i in range(4)]
        for proc in procs:
            proc.start()
        for proc in procs:
            proc.join()
    
    logger.warn('Initial free: {m}'.format(m = free_memory()))
    N = 15000
    data = np.ones((N,N))
    logger.warn('After allocating data: {m}'.format(m = free_memory()))
    
    if __name__ == '__main__':
        main()
    

which yielded

    
    
    [WARNING/MainProcess] Initial free: 2522340
    [WARNING/MainProcess] After allocating data: 763248
    [WARNING/Process-1] Free memory: 760852
    [WARNING/Process-2] Free memory: 757652
    [WARNING/Process-3] Free memory: 757264
    [WARNING/Process-4] Free memory: 756760
    

This shows that initially there was roughly 2.5GB of free memory. After
allocating a 15000x15000 array of `float64`s, there was 763248 KB free. This
roughly makes sense since 15000**2*8 bytes = 1.8GB and the drop in memory,
2.5GB - 0.763248GB is also roughly 1.8GB.

Now after each process is spawned, the free memory is again reported to be
~750MB. There is no significant decrease in free memory, so I conclude the
system must be using copy-on-write.

Conclusion: If you do not need to modify the data, defining it at the global
level of the `__main__` module is a convenient and (at least on Linux) memory-
friendly way to share it among subprocesses.

