python garbage collection after start process using multiprocess
Question: I test the gc behavior that python perform after start process using
multiprocess:

    
    
    from multiprocessing import Process
    import time
    
    class A(object):
        def __del__(self):
            print 'deleting'
    
    def f(name):
        import gc
        gc.collect()
        print 'hello', name
        print [map(lambda s: str(s)[:64], gc.get_referrers(o)) for o in gc.get_objects() if isinstance(o, A)]
        time.sleep(123)
    
    def main():
        a=A()
        p = Process(target=f, args=('bob',))
        p.start()
        p.join()
    
    
    if __name__ == '__main__':
        try:
            main()
        except:
            print 'sdfsdf!'
    

Output:

    
    
    hello bob
    [["[[], {'__setattr__': <slot wrapper '__setattr__' of 'object' obj", '<frame object at 0xb87570>', '<frame object at 0xbd7f80>']]
    

I want to close file descriptor by executing `__del__`. When the subprocess
starts, it enters the `f` function and the `A` instance `a` would no longer be
reachable. But the `__del__` is not executed so that means the `a` object is
still not freed. The output shows that it seems to be held by the frame
object.

So I tried another way using Exception to clean the stack to try to free the
unreachable object and execute `__del__` function:

    
    
    from multiprocessing import Process
    import time
    import sys
    
    class GcHelp(Exception):
        def __init__(self, func):
            self.func = func
            super(GcHelp, self).__init__(func.__name__)
    
    class A(object):
        def __del__(self):
            print 'deleting'
    
    def f():
        print 'target function'
    
    
    def raiser():
        raise GcHelp(f)
    
    def main():
        a=A()
        p = Process(target=raiser, args=())
        p.start()
        p.join()
    
    
    if __name__ == '__main__':
        try:
            main()
        except GcHelp as e:
            sys.exc_clear()
            e.func()
        except:
            print 'sdfsdf!' 
    

Output:

    
    
    Process Process-1:
    Traceback (most recent call last):
      File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
        self.run()
      File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
        self._target(*self._args, **self._kwargs)
      File "gc1.py", line 19, in raiser
        raise GcHelp(f)
    GcHelp: f
    

It seems that the multiprocess have ready clean the stack and take over all
exception handling.But parent frame does not exist any more. But why the frame
is still there in the first code example? Obviously it still holding the `a`
and the object is not freed at all.

Is there some way to perform this kind of gc in python?

Thanks a lot.


Answer: Why would you want to close the file? Assuming this is a linuxy system, a
forked environment is, well, forking weird. If you close the file in the
child, it will flush any data still in the buffers... but that same data will
be flushed again in the parent, resulting in duplicated data.

    
    
    import multiprocessing as mp
    
    fd = None
    
    def worker():
        # child closes file, flushing "1"
        fd.close()
    
    def doit():
        global fd
        fd = open('deleteme', 'w')
        fd.write('1')
        p = mp.Process(target=worker)
        p.start()
        p.join()
        # parent closes file, flushing "1"
        fd.close()
        # lets see what we got
        print(open('deleteme').read())
    
    doit()
    

This script prints `11` because the child and parent file objects both wrote
`1`. It gets far crazier if either side calls `flush` or `seek`.

_"it enters the f function and the A instance a would no longer be
reachable."_ That's not true in general. First, the only reason that the child
worker function exits the process when it returns is that the
`multiprocessing` module called it from a function that ensures exit. In the
general case, a forked function can return and execute its parent code. So,
from python's perspective, that `a` is still reachable. Further, your worker
could call a function that itself touches `a`. There is no way for python to
know that.

It would be incredibly expensive for python to clean up all "unreferencable"
objects after a fork. And incredibly dangerous as those objects can change the
system in many ways.

