Python - Mutliprocess, member functions of classes
Question: I can't figure out if this is because of me, or the multiprocessing module
that Python2.7 has. Can anyone figure out why this is not working?

    
    
    from multiprocessing import pool as mp
    class encapsulation:
       def __init__(self):
           self.member_dict = {}
       def update_dict(self,index,value):
           self.member_dict[index] = value
    encaps = encapsulation()
    def method(argument):
       encaps.update_dict(argument,argument)
       print encaps.member_dict
    p = mp() #sets up multiprocess pool of processors
    p.map(method,sys.argv[1:]) #method is the function, sys.argv is the list of arguments to multiprocess
    print encaps.member_dict
    >>>{argument:argument}
    >>>{}
    

So my question is just about member variables. It is my understanding that the
class encapsulation should hold this dictionary inside and outside of the
function. Why does it reset and give me an empty dictionary even though I have
only initialized it once? Please help


Answer: Even though you are encapsulating the object, the multiprocessing module will
end up using a local copy of the object in each process and never actually
propagate your changes back to you. In this case, you are not using the
Pool.map properly, as it expects each method call to return a result, which is
then sent back up to your return value. If what you want is to affect the
shared object, then you need a manager, which will coordinate the shared
memory:

## Encapsulating a shared object

    
    
    from multiprocessing import Pool 
    from multiprocessing import Manager
    import sys
    
    class encapsulation:
       def __init__(self):
           self.member_dict = {}
       def update_dict(self,index,value):
           self.member_dict[index] = value
    
    encaps = encapsulation()
    
    def method(argument):
       encaps.update_dict(argument,argument)
       # print encaps.member_dict       
    
    manager = Manager()
    encaps.member_dict = manager.dict()
    
    p = Pool()
    p.map(method,sys.argv[1:])
    
    print encaps.member_dict
    

**output**

    
    
    $ python mp.py a b c
    {'a': 'a', 'c': 'c', 'b': 'b'}
    

I would suggest not really setting the shared object as the member attribute,
but rather passing in as an arg, or encapsulating the shared object itself,
and then passing its values into your dict. The shared object cannot be kept
persistently. It needs to be emptied and discarded:

    
    
    # copy the values to a reg dict
    encaps.member_dict = encaps.member_dict.copy()
    

But this might even be better:

    
    
    class encapsulation:
       def __init__(self):
           self.member_dict = {}
       # normal dict update
       def update_dict(self,d):
           self.member_dict.update(d)
    
    encaps = encapsulation()
    
    manager = Manager()
    results_dict = manager.dict()
    
    # pass in the shared object only
    def method(argument):
       results_dict[argument] = argument    
    
    p = Pool()
    p.map(method,sys.argv[1:])
    
    encaps.update_dict(results_dict)
    

## Using the pool.map as intended

If you were using the map to return values, it might look like this:

    
    
    def method(argument):
       encaps.update_dict(argument,argument)
       return encaps.member_dict
    
    p = Pool()
    results = p.map(method,sys.argv[1:]) 
    print results
    # [{'a': 'a'}, {'b': 'b'}, {'c': 'c'}]
    

You would need to combine the results into your dict again:

    
    
    for result in results:
        encaps.member_dict.update(result)
    print encaps.member_dict
    # {'a': 'a', 'c': 'c', 'b': 'b'}
    

