Pickling method descriptor objects in python
Question: I am trying to pickle a `method_descriptor`.

  

Pickling with `pickle` or `cloudpickle` fails:

    
    
    Python 2.7.10 |Continuum Analytics, Inc.| (default, Oct 19 2015, 18:04:42) 
    [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    Anaconda is brought to you by Continuum Analytics.
    Please check out: http://continuum.io/thanks and https://anaconda.org
    
    >>> import pickle, cloudpickle
    
    >>> pickle.dumps(set.union)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/home/pmd/anaconda3/envs/python2/lib/python2.7/pickle.py", line 1374, in dumps
        Pickler(file, protocol).dump(obj)
      File "/home/pmd/anaconda3/envs/python2/lib/python2.7/pickle.py", line 224, in dump
        self.save(obj)
      File "/home/pmd/anaconda3/envs/python2/lib/python2.7/pickle.py", line 306, in save
        rv = reduce(self.proto)
      File "/home/pmd/anaconda3/envs/python2/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
        raise TypeError, "can't pickle %s objects" % base.__name__
    TypeError: cannot pickle method_descriptor objects
    
    >>> cloudpickle.dumps(set.union)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/home/pmd/anaconda3/envs/python2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 602, in dumps
        cp.dump(obj)
      File "/home/pmd/anaconda3/envs/python2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 111, in dump
        raise pickle.PicklingError(msg)
    pickle.PicklingError: Could not pickle object as excessively deep recursion required.
    

  

Importing `dill` somehow makes `pickle` work, as shown below:

    
    
    >>> import dill
    >>> pickle.dumps(set.union)
    'cdill.dill\n_getattr\np0\n(c__builtin__\nset\np1\nS\'union\'\np2\nS"<method \'union\' of \'set\' objects>"\np3\ntp4\nRp5\n.'
    
    >>> f = pickle.loads(pickle.dumps(set.union))
    >>> set.union(set([1,2]), set([3]))
    set([1, 2, 3])
    >>> f(set([1,2]), set([3]))
    set([1, 2, 3])
    

  

The issue in `cloudpickle` remains even after the `dill` import:

    
    
    >>> cloudpickle.dumps(set.union)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/home/pmd/anaconda3/envs/python2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 602, in dumps
        cp.dump(obj)
      File "/home/pmd/anaconda3/envs/python2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 111, in dump
        raise pickle.PicklingError(msg)
    pickle.PicklingError: Could not pickle object as excessively deep recursion required.
    

  

In my application I rely on `cloudpickle` to handle functions with globals. So
my question is, how can I get `cloudpickle` to work for `method_descriptor`
objects in Python 2.7?

EDIT: I noticed that the same issue occurs in Python 3.3, but is not present
in Python 3.5.


Answer: I'm the `dill` author. When you do `import dill`, it injects the serialization
registry from `dill` into `pickle` (basically, puts all the `copy_reg`-type
knowledge from `dill` into the `pickle` registry).

    
    
    >>> import pickle
    >>> pickle.Pickler.dispatch
    {<type 'function'>: <function save_global at 0x105d0c7d0>, <type 'dict'>: <function save_dict at 0x105d0c668>, <type 'int'>: <function save_int at 0x105d0c230>, <type 'long'>: <function save_long at 0x105d0c2a8>, <type 'list'>: <function save_list at 0x105d0c578>, <type 'str'>: <function save_string at 0x105d0c398>, <type 'unicode'>: <function save_unicode at 0x105d0c410>, <type 'instance'>: <function save_inst at 0x105d0c758>, <type 'type'>: <function save_global at 0x105d0c7d0>, <type 'NoneType'>: <function save_none at 0x105d0c140>, <type 'bool'>: <function save_bool at 0x105d0c1b8>, <type 'tuple'>: <function save_tuple at 0x105d0c488>, <type 'float'>: <function save_float at 0x105d0c320>, <type 'classobj'>: <function save_global at 0x105d0c7d0>, <type 'builtin_function_or_method'>: <function save_global at 0x105d0c7d0>}
    >>> import dill
    >>> pickle.Pickler.dispatch
    {<class '_pyio.BufferedReader'>: <function save_file at 0x106c8b848>, <class '_pyio.TextIOWrapper'>: <function save_file at 0x106c8b848>, <type 'operator.itemgetter'>: <function save_itemgetter at 0x106c8b578>, <type 'weakproxy'>: <function save_weakproxy at 0x106c8c050>, <type 'NoneType'>: <function save_none at 0x105d0c140>, <type 'str'>: <function save_string at 0x105d0c398>, <type 'file'>: <function save_file at 0x106c8b8c0>, <type 'classmethod'>: <function save_classmethod at 0x106c8c230>, <type 'float'>: <function save_float at 0x105d0c320>, <type 'instancemethod'>: <function save_instancemethod0 at 0x106c8ba28>, <type 'cell'>: <function save_cell at 0x106c8bb18>, <type 'member_descriptor'>: <function save_wrapper_descriptor at 0x106c8bc08>, <type 'slice'>: <function save_slice at 0x106c8bc80>, <type 'dict'>: <function save_module_dict at 0x106c8b410>, <type 'long'>: <function save_long at 0x105d0c2a8>, <type 'code'>: <function save_code at 0x106c8b320>, <type 'type'>: <function save_type at 0x106c8c0c8>, <type 'xrange'>: <function save_singleton at 0x106c8bde8>, <type 'builtin_function_or_method'>: <function save_builtin_method at 0x106c8b9b0>, <type 'classobj'>: <function save_classobj at 0x106c8b488>, <type 'weakref'>: <function save_weakref at 0x106c8bed8>, <type 'getset_descriptor'>: <function save_wrapper_descriptor at 0x106c8bc08>, <type 'weakcallableproxy'>: <function save_weakproxy at 0x106c8c050>, <class '_pyio.BufferedRandom'>: <function save_file at 0x106c8b848>, <type 'int'>: <function save_int at 0x105d0c230>, <type 'list'>: <function save_list at 0x105d0c578>, <type 'functools.partial'>: <function save_functor at 0x106c8b7d0>, <type 'bool'>: <function save_bool at 0x105d0c1b8>, <type 'function'>: <function save_function at 0x106c8b398>, <type 'thread.lock'>: <function save_lock at 0x106c8b500>, <type 'super'>: <function save_functor at 0x106c8b938>, <type 'staticmethod'>: <function save_classmethod at 0x106c8c230>, <type 'module'>: <function save_module at 0x106c8bf50>, <type 'method_descriptor'>: <function save_wrapper_descriptor at 0x106c8bc08>, <type 'operator.attrgetter'>: <function save_attrgetter at 0x106c8b5f0>, <type 'wrapper_descriptor'>: <function save_wrapper_descriptor at 0x106c8bc08>, <type 'numpy.ufunc'>: <function save_numpy_ufunc at 0x106c8bcf8>, <type 'method-wrapper'>: <function save_instancemethod at 0x106c8baa0>, <type 'instance'>: <function save_inst at 0x105d0c758>, <type 'cStringIO.StringI'>: <function save_stringi at 0x106c8b6e0>, <type 'unicode'>: <function save_unicode at 0x105d0c410>, <class '_pyio.BufferedWriter'>: <function save_file at 0x106c8b848>, <type 'property'>: <function save_property at 0x106c8c140>, <type 'ellipsis'>: <function save_singleton at 0x106c8bde8>, <type 'tuple'>: <function save_tuple at 0x105d0c488>, <type 'cStringIO.StringO'>: <function save_stringo at 0x106c8b758>, <type 'NotImplementedType'>: <function save_singleton at 0x106c8bde8>, <type 'dictproxy'>: <function save_dictproxy at 0x106c8bb90>}
    

`cloudpickle` has (slightly) different pickling functions than `dill`, and if
you are using `cloudpickle`, it push it's own serialization functions into the
`pickle` registry. If you want to get `cloudpickle` to work for you, you might
be able to monkeypatch a solution… essentially install a module within your
application that does `import dill as cloudpickle` (Nice reference:
<http://blog.dscpl.com.au/2015/03/safely-applying-monkey-patches-in-
python.html>)… but that would replace the entire use of `cloudpickle` with
`dill` in your application context. You could also try a monkeypatch along
these lines:

    
    
    >>> #first import dill, which populates itself into pickle's dispatch
    >>> import dill
    >>> import pickle
    >>> # save the MethodDescriptorType from dill
    >>> MethodDescriptorType = type(type.__dict__['mro'])
    >>> MethodDescriptorWrapper = pickle.Pickler.dispatch[MethodDescriptorType]
    >>> # cloudpickle does the same, so let it update the dispatch table
    >>> import cloudpickle
    >>> # now, put the saved MethodDescriptorType back in
    >>> pickle.Pickler.dispatch[MethodDescriptorWrapperType] = MethodDescriptorWrapper
    

Note that if you are going to use `cloudpickle.dumps` directly, you'd have to
overload the registry in `cloudpickle` directly by doing the above monkeypatch
on `cloudpickle.CloudPickler.dispatch`.

I don't guarantee that it _will_ work, nor do I guarantee that it won't screw
up other objects from `cloudpickle` (essentially, I haven't tried it), but
it's a potential route to replacing the offending `cloudpickle` wrapper with
the one from `dill`.

If you want the short answer, I'd say (at least for this case) use `dill`. ;)

* * *

**EDIT** with regard to `copyreg`:

Here's what's in `dill`:

    
    
    def _getattr(objclass, name, repr_str):
        # hack to grab the reference directly
        try:
            attr = repr_str.split("'")[3]
            return eval(attr+'.__dict__["'+name+'"]')
        except:
            attr = getattr(objclass,name)
            if name == '__dict__':
                attr = attr[name]
            return attr
    

Which is used to register a function with a lower-level reduce function
(directly on the pickler instance). `obj` is the object to pickle.

    
    
    pickler.save_reduce(_getattr, (obj.__objclass__, obj.__name__, obj.__repr__()), obj=obj)
    

I believe this translates to a reduce method (used directly in
`copyreg.pickle`) like this:

    
    
    def _reduce_method_descriptor(obj):
        return _getattr, (obj.__objclass__, obj.__name__, obj.__repr__())
    

