Shared memory Value dissapears from an object
Question: here is some strange behavior in Python (2.7.9, Windows) I would like to ask
an advice about.

I am trying to extend a class with a variable in initialization and I want to
share an object of this class between two processes (parent and child). Both
of them will use this variable.

This code works fine:

    
    
    # coding: utf-8
    from multiprocessing import Process, Value
    from time import sleep
    
    
    class ExQueue(object):
        def __init__(self, *args, **kwargs):
            super(ExQueue, self).__init__(*args, **kwargs)
            self.trust_me_this_name_is_unique = Value('L', 0)
    
    
    def func(ex_queue):
        print 'I am the child process. Is there a parameter: ',
        print hasattr(ex_queue, 'trust_me_this_name_is_unique')
    
        # this sleep is here to assure no printing overlapping
        sleep(.5)
    
    def main():
        ex_queue = ExQueue()
    
        print 'I am the parent process. Is there a parameter: ',
        print hasattr(ex_queue, 'trust_me_this_name_is_unique')
    
        child_process = Process(target=func, args=(ex_queue,))
    
        child_process.start()
    
        child_process.join()
    
        print 'I am the parent process. Is there a parameter: ',
        print hasattr(ex_queue, 'trust_me_this_name_is_unique')
    
    if __name__ == '__main__':
        main()
    

Output:

    
    
    I am the parent process. Is there a parameter:  True
    I am the child process. Is there a parameter:  True
    I am the parent process. Is there a parameter:  True
    

But if `ExQueue` is inherited from `Queue` class (the one from
`multiprocessing.queues` module), the trick does not work.

At the beginning add:

    
    
    from multiprocessing.queues import Queue
    

and change

    
    
    class ExQueue(object):
    

to:

    
    
    class ExQueue(Queue):
    

Output will be:

    
    
    I am the parent process. Is there a parameter:  True
    I am the child process. Is there a parameter:  False
    I am the parent process. Is there a parameter:  True
    

So, in the `ExQueue` example of the child process there is no
`trust_me_this_name_is_unique` variable.

Any ideas on what I am doing wrong?

Thanks!

**UPD:** Works as expected on Mac OS. Variable does not disappear.

## Solution

Thank you, **tdelaney**! Your answer helped a lot!

I added those two methods to the `ExQueue` and it pickles on Windows just fine
now:

    
    
    def __getstate__(self):
        state = super(ExQueue, self).__getstate__()
        return state + (self.trust_me_this_name_is_unique,)
    
    def __setstate__(self, state):
        state, self.trust_me_this_name_is_unique = state[:-1], state[-1]
        super(ExQueue, self).__setstate__(state)
    

Still not sure that it's a good approach to inherit from `Queue` at all. :)


Answer: Its not a problem with the shared memory value, its the variable on the queue
object itself that disappears.

Multiprocessing works differently on machines that implement the *nix forking
model verses the Windows create-process model. On *nix, when you create a
`multiprocessing` process, the parent process is forked and since the child
has as copy-on-write view of the parent's memory space, all of the python
objects (including your queue) are in the child space ready to be used.

On Windows, there is no `fork`. A new process is created and the relevant
parts of the parent process is pickled, sent to the child and unpickled. This
only works for picklable objects so code that works in linux may fail in
windows.

If you have an object that isn't natively picklable, you can implement
`__getstate__` and `__setstate__` methods that return a pickable subset of the
object and rebuild the object from that state. That's been done with the
`multiprocessing.Queue` object. It doesn't include your variable in
`__getstate__` so the variable isn't included in the child object when its
recreated.

An easy solution is to put your data some place else. If that isn't viable,
create your own subclass of `multiprocessing.Queue` and write your own
`__getstate__` and `__setstate__` methods.

