PicklingError: Can't pickle <type 'function'> with python process pool executor
Question: **util.py**

    
    
    def exec_multiprocessing(self, method, args):
        with concurrent.futures.ProcessPoolExecutor() as executor:
            results = pool.map(method, args)
            return results
    

**clone.py**

    
    
    def clone_vm(self, name, first_run, host, ip):
        # clone stuff
    

**invoke.py**

    
    
    exec_args = [(name, first_run, host, ip) for host, ip in zip(hosts, ips)]
    results = self.util.exec_multiprocessing(self.clone.clone_vm, exec_args)
    

The above code gives the pickling error. I found that it is because we are
passing instance method. So we should unwrap the instance method. But I am not
able to make it work.

Note: I can not create top level method to avoid this. I have to use instance
methods.


Answer: Let's start with an overview - why the error came up in the first place:

The `multiprocessing` must requires to pickle (serialize) data to pass them
along processes or threads. To be specific, `pool` methods themselves rely on
`queue` at the lower level, to stack tasks and pass them to threads/processes,
and `queue` requires everything that goes through it must be pickable.

The problem is, not all items are pickable - [list of
pickables](https://docs.python.org/2/library/pickle.html#what-can-be-pickled-
and-unpickled) \- and when one tries to pickle an unpicklable object, gets the
**`PicklingError`** exception - exactly what happened in your case, you passed
an instance method which is not picklable.

There can be various workarounds (as is the case with every problem) - the
solution which worked for me is [here by
Dano](http://stackoverflow.com/q/25156768/2073595) \- is to make `pickle`
handle the methods and register it with
[`copy_reg`](https://docs.python.org/2/library/copy_reg.html).

* * *

Add the following lines at the start of your module `clone.py` to make
`clone_vm` picklable (do `import` `copy_reg` and `types`):

    
    
    def _pickle_method(m):
        if m.im_self is None:
            return getattr, (m.im_class, m.im_func.func_name)
        else:
            return getattr, (m.im_self, m.im_func.func_name)
    
    copy_reg.pickle(types.MethodType, _pickle_method)
    

* * *

Other useful answers - by [Alex
Martelli](http://stackoverflow.com/questions/1816958/cant-pickle-type-
instancemethod-when-using-pythons-multiprocessing-pool-ma/1816969#1816969),
[mrule](http://stackoverflow.com/questions/3288595/multiprocessing-using-pool-
map-on-a-function-defined-in-a-class), by
[unutbu](http://stackoverflow.com/questions/8804830/python-multiprocessing-
pickling-error)

