Decorate instance method with (arbitrary) meta data in python Requirements

What I need: attach meta information to methods, such that


it is 'easy' to retrieve all such methods for a given class instance
methods can still be called 'in a normal way', e.g. obj.method()
meta-data is accessible from the decorated method, e.g. obj.method.data
IDEs (PyCharm in particular) do not produce any warnings or errors (and, if possible, IDE support, e.g. auto-completion, should be able to handle the annotation)


Additionally, I would like the code to be readable/intuitive (not necessarily the super classes, though), generally robust and 'bug-free'. I accept the limitation that my decorators need to be the most 'outer' decorator for the automatic collection to take place.

From my point of view, overcoming function/method transformation while still exposing an arbitrary object type (not a function type -- thinking of this, maybe subclassing a FunctionType might be another idea?) is the hardest challenge.

What do you think of the following three solutions? Is there something I did miss?

Code

class MethodDecoratorWithIfInCall(object):
    def __init__(self):
        self._func = None

    def __call__(self, *args, **kwargs):
        if self._func is None:
            assert len(args) == 1 and len(kwargs) == 0
            self._func = args[0]
            return self
        else:
            return self._func(*args, **kwargs)

    def __get__(self, *args, **kwargs):
        # update func reference to method object
        self._func = self._func.__get__(*args, **kwargs)
        return self


class MacroWithIfInCall(MethodDecoratorWithIfInCall):
    def __init__(self, name):
        super(MacroWithIfInCall, self).__init__()
        self.name = name


class MethodDecoratorWithExplicitDecorate(object):
    def __init__(self, *args, **kwargs):
        # wildcard parameters to satisfy PyCharm
        self._func = None

    def __call__(self, *args, **kwargs):
        return self._func(*args, **kwargs)

    def __get__(self, *args, **kwargs):
        # update func reference to method object
        self._func = self._func.__get__(*args, **kwargs)
        return self

    def _decorate(self):
        def _set_func(func):
            self._func = func
            return self
        return _set_func

    @classmethod
    def decorate(cls, *args, **kwargs):
        obj = cls(*args, **kwargs)
        return obj._decorate()


class MacroWithExplicitDecorate(MethodDecoratorWithExplicitDecorate):
    def __init__(self, name):
        super(MacroWithExplicitDecorate, self).__init__()
        self.name = name


class MacroWithoutSuperclass(object):
    def __init__(self, func, name):
        self.func = func
        self.name = name

    def __get__(self, *args, **kwargs):
        # update func reference to method object
        self.func = self.func.__get__(*args, **kwargs)
        return self

    def __call__(self, *args, **kwargs):
        return self.func(*args, **kwargs)

    @staticmethod
    def decorate(name):
        return lambda func: MacroWithoutSuperclass(func, name)


class Shell:
    def __init__(self):
        macros = [macro for macro in map(self.__getattribute__, dir(self))
                  if isinstance(macro, (MacroWithIfInCall, MacroWithExplicitDecorate, MacroWithoutSuperclass))]

        for macro in macros:
            print(macro.name, macro())

    @MacroWithIfInCall(name="macro-with-if-in-call")
    def macro_add_1(self):
        return "called"

    @MacroWithExplicitDecorate.decorate(name="macro-with-explicit-decorate")
    def macro_add_2(self):
        return "called"

    @MacroWithoutSuperclass.decorate(name="macro-without-superclass")
    def macro_add_3(self):
        return "called"


if __name__ == '__main__':
    shell = Shell()


Output

macro-with-if-in-call called
macro-with-explicit-decorate called
macro-without-superclass called
From your three approaches, I think MacroWithoutSuperclass is probably the cleanest. I wanted to just comment a bit, but it turned out I had too much to say... Thus, here a few remarks followed by a maybe more intuitive solution as an inspiration (it's probably not perfect).



None of your methods supports arbitrary meta data, in case that is a requirement, but only name. In the following I assume that you in fact want to add arbitrary data.

You are assigning meta-data by name, so that e.g. macro.name is accessible. Is that a requirement? Or would macro.meta['name'] be enough? Below I will present both options, but it is not really clear from you question, only implicitly from your code.

All of your three methods seem in general very complicated.
For example

if self._func is None:
    assert len(args) == 1 and len(kwargs) == 0
    self._func = args[0]
    return self
else:
    return self._func(*args, **kwargs)


is quite complicated and should have some more explanations.

The inheritance

class MacroWithIfInCall(MethodDecoratorWithIfInCall):
    def __init__(self, name):
        super(MacroWithIfInCall, self).__init__()
        self.name = name


is not needed (the same applies to MacroWithExplicitDecorate), instead you can assign the name in MethodDecoratorWithIfInCall's init function – exactly as you did in the MacroWithoutSuperclass. Which is basically why I think a combination of the MacroWithIfInCall (the __call__ function) and the MacroWithoutSuperclass would be the best solution using your approach.

I don't understand why you need wildcard parameters (# wildcard parameters to satisfy PyCharm) for one class, but not for the others, you shouldn't need them in either case.

Anyways, I think you can fare much better with an even simpler decorator, as I will explain below.



I will try to go along your list and provide some comments to answer your requirements.


  
  it is 'easy' to retrieve all such methods for a given class instance
  


This is sufficiently easy using inspect.getmembers:

import inspect

types = (MacroWithIfInCall, MacroWithExplicitDecorate, MacroWithoutSuperclass)
macros = inspect.getmembers(Shell, lambda m: isinstance(m, types))


Note however, that macros is now a list containing tuples of the form (name, function). Dependending on your use case you might have to use any of the following lines:

names = [macro[0] for macro in macros]  # Get only the names
funcs = [macro[1] for macro in macros]  # Get only the functions
names, funcs = zip(*macros)  # Get names and functions in separate lists



  
  methods can still be called 'in a normal way', e.g. obj.method()
  


This is fine, and will be the case for the classic decorator which follows this form:

import functools

def decorator(func):

    @functools.wraps(func)  # Handles the docs and names properly
    def wrapper(*args, **kwargs):
        # Do something before the call

        result = func(*args, **kwargs)

        # Do something after the call

        return result

    return wrapper


In fact, for your problem slightly less complex, as you don't need to perform any tasks on execution but only when declaring the function, so in fact you can return the original function:

def meta(func):
    func.meta = 'meta data here'  # Assign constant metadata
    return func



  
  meta-data is accessible from the decorated method, e.g. obj.method.data
  


As mentioned above, I am not sure whether you mean that all meta data is contained in data or that data is just one part, and name, age, abc would be others.

To have this, you could now just assign these values by extending the decorator meta(func) I presented above. Remember that in Python, functions are just objects (search for "User-defined functions"):


  Function objects also support getting and setting arbitrary attributes, which can be used, for example, to attach metadata to functions. Regular attribute dot-notation is used to get and set such attributes.


This means, that instead of having your complex wrapper classes storing the meta data and the function you can just store the meta data at the function itself.

So the decorator meta needs to have a way to specify the meta data. The common way is building another function around it, which eventually returns the decorator.

def meta(**meta_data):
    """Attaches meta information to a method."""

    def _attach_meta(func):
        func.meta = meta_data
        return func

    return _attach_meta


To support the dot-notation (and not the meta-dictionary) the line func.meta = meta_data could to be changed to 

for k, v in meta_data.items():
    setattr(func, k, v)
func.has_meta = True  # flag to use for inspect.getmembers' predicate


I would use the dictionary version unless you have a very strict set of meta attributes, as then accessing data using function.meta.get(KEY, DEFAULT_VALUE) becomes fairly useful.


  
  IDEs (PyCharm in particular) do not produce any warnings or errors (and, if possible, IDE support, e.g. auto-completion, should be able to handle the annotation)
  


I can't say anything about this, but I think it should work as you expect... I use YCM and am too lazy to install PyCharm now.


  Additionally, I would like the code to be readable/intuitive (not necessarily the super classes, though)


This should be sufficiently satisfied with the presented solution, although it is also lacking some (in-code) documentation.


  generally robust and 'bug-free'. 


There can always be name clashes etc., and my solution does not have the strong typing your solution has. But for bug-free-ness, it is possible to add some tests, making it less buggy.


  I accept the limitation that my decorators need to be the most 'outer' decorator for the automatic collection to take place.


This should still be the case, I fear. But I haven't tried to solve this.



Full working example

import inspect


def meta(**meta_data):
    """Attaches meta information to a method."""

    def _attach_meta(func):
        func.meta = meta_data
        return func

    return _attach_meta


def has_meta(func):
    """Predicate to check for meta data."""
    return hasattr(func, 'meta')


class SomeAnnotated:
    @meta(a=1, b=2)
    def __call__(self):
        """The call method."""
        return 'Hello'

    @meta(x=3, y=4)
    def annotated(self, abc):
        """The annotated method."""
        return abc

    def not_annotated(self):
        """The not annotated method."""
        return 'Not annotated'


print('Meta information is available for:')
for name, func in inspect.getmembers(SomeAnnotated, has_meta):
    print(f'{func.__qualname__}: {func.meta} -- {func.__doc__}')


Expected output:

Meta information is available for:
SomeAnnotated.__call__: {'a': 1, 'b': 2} -- The call method.
SomeAnnotated.annotated: {'x': 3, 'y': 4} -- The annotated method.




Edit: As per comment here is a solution more closely related to your idea around a non-data descriptor (i.e. using __get__):

import inspect
import functools


class MetaAnnotated(functools.partial):
    pass


class MetaAnnotator:
    def __init__(self, func, **meta_data):
        self.func = func
        self.meta = meta_data

    def __get__(self, obj, type=None):
        annotated_func = functools.wraps(self.func)(MetaAnnotated(self.func, obj))
        annotated_func.meta = self.meta
        return annotated_func


def meta(**meta_data):
    """Attaches meta information to a method."""

    def _attach_meta(func):
        return MetaAnnotator(func, **meta_data)

    return _attach_meta


def has_meta(func):
    """Predicate to check for meta data."""
    return isinstance(func, MetaAnnotated)


The above solution uses functools for two purposes: first, to retain the documentation etc (wraps). Second, and more importantly, to create a partial function which is called with our instance as self. In a step-by-step solution we would do this as follows (each block represents a change to understand the code, not a sequence of commands):

# annotated_func behaves the same as func, but will 
# always use "obj" as its first argument.
annotated_func = functools.partial(func, obj)

# We can inherit from functools.partial to have a tagging class:
class MetaAnnotated(functools.partial): pass
annotated_func = MetaAnnotated(func, obj)

# Keep the doc string from func:
class MetaAnnotated(functools.partial): pass
annotated_func = functools.wraps(func)(MetaAnnotated(func, obj))


This way, no __call__ is needed, neither is a staticmethod: as soon as the function we decorated using MetaAnnotated is searched for on the original object, the properly wrapped function is returned (instead of the MetaAnnotator!). 

This behaves almost the same as the solution without typing, but it does not have the func.__qualname__ needed in my test prints, so we need to change this. I also added a "proof" that it's properly callable:

print('Meta information is available for:')
for name, func in inspect.getmembers(SomeAnnotated, has_meta):
    print(f'{name}: {func.meta} -- {func.__doc__}')

instance = SomeAnnotated()
print(instance())
print(instance.annotated(123))


I think this is a nice synthesis between your original ideas about using the descriptors and my ideas of having a very plain and straight forward decorator. All the "magic" happens in one place (the MetaAnnotator), the MetaAnnotated class transparently tags functions, and there is a very neat API: a user just needs to call @meta and can forget about everything else.