Why is multiprocessing copying my data if I don't touch it?
Question: I was tracking down an out of memory bug, and was horrified to find that
python's multiprocessing appears to copy large arrays, even if I have no
intention of using them.

Why is python (on Linux) doing this, I thought copy-on-write would protect me
from any extra copying? I imagine that whenever I reference the object some
kind of trap is invoked and only then is the copy made.

Is the correct way to solve this problem for an arbitrary data type, like a 30
gigabyte custom dictionary to use a `Monitor`? Is there some way to build
Python so that it doesn't have this nonsense?

    
    
    import numpy as np
    import psutil
    from multiprocessing import Process
    mem=psutil.virtual_memory()
    large_amount=int(0.75*mem.available)
    
    def florp():
        print("florp")
    
    def bigdata():
        return np.ones(large_amount,dtype=np.int8)
    
    if __name__=='__main__':
        foo=bigdata()#Allocated 0.75 of the ram, no problems
        p=Process(target=florp)
        p.start()#Out of memory because bigdata is copied? 
        print("Wow")
        p.join()
    

Running:

    
    
    [ebuild   R    ] dev-lang/python-3.4.1:3.4::gentoo  USE="gdbm ipv6 ncurses readline ssl threads xml -build -examples -hardened -sqlite -tk -wininst" 0 KiB
    


Answer: I'd expect this behavior -- when you pass code to Python to compile, anything
that's not guarded behind a function or object is immediately `exec`ed for
evaluation.

In your case, `bigdata=np.ones(large_amount,dtype=np.int8)` has to be
evaluated -- unless your actual code has different behavior, `florp()` not
being called has nothing to do with it.

To see an immediate example:

    
    
    >>> f = 0/0
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ZeroDivisionError: integer division or modulo by zero
    >>> def f():
    ...     return 0/0
    ...
    >>>
    

To apply this to your code, put `bigdata=np.ones(large_amount,dtype=np.int8)`
behind a function and call it as your need it, otherwise, Python is trying to
be hepful by having that variable available to you at runtime.

If `bigdata` doesn't change, you could write a function that gets or sets it
on an object that you keep around for the duration of the process.

edit: Coffee just started working. When you make a new process, Python will
need to copy all objects into that new process for access. You can avoid this
by using threads or by a mechanism that will allow you to share memory between
processes such as [shared memory
maps](https://docs.python.org/2/library/multiprocessing.html#sharing-state-
between-processes) or shared
[ctypes](https://docs.python.org/2/library/multiprocessing.html#module-
multiprocessing.sharedctypes)

