How can I restrict the scope of a multiprocessing process?
Question: Using python's
[`multiprocessing`](https://docs.python.org/2/library/multiprocessing.html)
module, the following contrived example runs with minimal memory requirements:

    
    
    import multiprocessing 
    # completely_unrelated_array = range(2**25)
    
    def foo(x):
        for x in xrange(2**28):pass
        print x**2
    
    P = multiprocessing.Pool()
    
    for x in range(8):
        multiprocessing.Process(target=foo, args=(x,)).start()
    

Uncomment the creation of the `completely_unrelated_array` and you'll find
that each spawned process allocates the memory for a copy of the
`completely_unrelated_array`! This is a minimal example of a much larger
project that I can't figure out how to workaround; multiprocessing seems to
make a copy of everything that is global. I **don't** need a shared memory
object, I simply need to pass in `x`, and process it _without_ the memory
overhead of the entire program.

_Side observation_ : What's interesting is that `print
id(completely_unrelated_array)` inside `foo` gives the same value, suggesting
that somehow that might not be copies...


Answer: Because of the nature of `os.fork()`, any variables in the global namespace of
your `__main__` module will be inherited by the child processes (assuming
you're on a Posix platform), so you'll see the memory usage in the children
reflect that as soon as they're created. I'm not sure if all that memory is
really being allocated though, as far as I know that memory is shared until
you actually try to change it in the child, at which point a new copy is made.
Windows, on the other hand, doesn't use `os.fork()` \- it re-imports the main
module in each child, and pickles any local variables you want sent to the
children. So, using Windows you can actually avoid the large global ending up
copied in the child by only defining it inside an `if __name__ == "__main__":`
guard, because everything inside that guard will only run in the parent
process:

    
    
    import time
    import multiprocessing 
    
    
    def foo(x):
        for x in range(2**28):pass
        print(x**2)
    
    if __name__ == "__main__":
        completely_unrelated_array = list(range(2**25)) # This will only be defined in the parent on Windows
        P = multiprocessing.Pool()
    
        for x in range(8):
            multiprocessing.Process(target=foo, args=(x,)).start()
    

Now, in Python 2.x, you can only create new `multiprocessing.Process` objects
by forking if you're using a Posix platform. But on Python 3.4, you can
specify how the new processes are created, by using contexts. So, we can
specify the
[`"spawn"`](https://docs.python.org/3/library/multiprocessing.html?highlight=multiprocessing#contexts-
and-start-methods) context, which is the one Windows uses, to create our new
processes, and use the same trick:

    
    
    # Note that this is Python 3.4+ only
    import time
    import multiprocessing 
    
    def foo(x):
        for x in range(2**28):pass
        print(x**2)
    
    
    if __name__ == "__main__":
        completely_unrelated_array = list(range(2**23))  # Again, this only exists in the parent
        ctx = multiprocessing.get_context("spawn") # Use process spawning instead of fork
        P = ctx.Pool()
    
        for x in range(8):
            ctx.Process(target=foo, args=(x,)).start()
    

If you need 2.x support, or want to stick with using `os.fork()` to create new
`Process` objects, I think the best you can do to get the reported memory
usage down is immediately delete the offending object in the child:

    
    
    import time
    import multiprocessing 
    import gc
    
    def foo(x):
        init()
        for x in range(2**28):pass
        print(x**2)
    
    def init():
        global completely_unrelated_array
        completely_unrelated_array = None
        del completely_unrelated_array
        gc.collect()
    
    if __name__ == "__main__":
        completely_unrelated_array = list(range(2**23))
        P = multiprocessing.Pool(initializer=init)
    
        for x in range(8):
            multiprocessing.Process(target=foo, args=(x,)).start()
        time.sleep(100)
    

