Python: Idiomatic properties for structured data?
Question: I've got a bad smell in my code. Perhaps I just need to let it air out for a
bit, but right now it's bugging me.

I need to create three different input files to run three Radiative Transfer
Modeling (RTM) applications, so that I can compare their outputs. This process
will be repeated for thousands of sets of inputs, so I'm automating it with a
python script.

I'd like to store the input parameters as a generic python object that I can
pass to three other functions, who will each translate that general object
into the specific parameters needed to run the RTM software they are
responsible. I think this makes sense, but feel free to criticize my approach.

There are many possible input parameters for each piece of RTM software. Many
of them over-lap. Most of them are kept at sensible defaults, but should be
easily changed.

I started with a simple `dict`

    
    
    config = {
        day_of_year: 138,
        time_of_day: 36000, #seconds
        solar_azimuth_angle: 73, #degrees
        solar_zenith_angle: 17, #degrees
        ...
    }
    

There are a lot of parameters, and they can be cleanly categorized into
groups, so I thought of using `dict`s within the `dict`:

    
    
    config = {
        day_of_year: 138,
        time_of_day: 36000, #seconds
        solar: {
            azimuth_angle: 73, #degrees
            zenith_angle: 17, #degrees
            ...
        },
        ...
    }
    

I like that. But there are a lot of redundant properties. The solar azimuth
and zenith angles, for example, can be found if the other is known, so why
hard-code both? So I started looking into python's builtin
[`property`](http://docs.python.org/library/functions.html#property). That
lets me do nifty things with the data if I store it as object attributes:

    
    
    class Configuration(object):
        day_of_year = 138,
        time_of_day = 36000, #seconds
        solar_azimuth_angle = 73, #degrees
        @property
        def solar_zenith_angle(self):
            return 90 - self.solar_azimuth_angle
        ...
    
    config = Configuration()
    

But now I've lost the structure I had from the second `dict` example.

Note that some of the properties are less trivial than my `solar_zenith_angle`
example, and might require access to other attributes outside of the group of
attributes it is a part of. For example I can calculate `solar_azimuth_angle`
if I know the day of year, time of day, latitude, and longitude.

**What I'm looking for:**

A simple way to store configuration data whose values can all be accessed in a
uniform way, are nicely structured, and may exist either as attributes (real
values) or properties (calculated from other attributes).

**A possibility that is kind of boring:**

Store everything in the dict of dicts I outlined earlier, and having other
functions run over the object and calculate the calculatable values? This
doesn't sound fun. Or clean. To me it sounds messy and frustrating.

**An ugly one that works:**

After a long time trying different strategies and mostly getting no where, I
came up with one possible solution that seems to work:

**My classes:** (smells a bit func-y, er, funky. def-initely.)

    
    
    class SubConfig(object):
        """
        Store logical groupings of object attributes and properties.
    
        The parent object must be passed to the constructor so that we can still
        access the parent object's other attributes and properties. Useful if we
        want to use them to compute a property in here.
        """
        def __init__(self, parent, *args, **kwargs):
            super(SubConfig, self).__init__(*args, **kwargs)
            self.parent = parent
    
    
    class Configuration(object):
        """
        Some object which holds many attributes and properties.
    
        Related configurations settings are grouped in SubConfig objects.
        """
        def __init__(self, *args, **kwargs):
            super(Configuration, self).__init__(*args, **kwargs)
            self.root_config = 2
    
            class _AConfigGroup(SubConfig):
                sub_config = 3
                @property
                def sub_property(self):
                    return self.sub_config * self.parent.root_config
            self.group = _AConfigGroup(self) # Stinky?!
    

**How I can use them:** (works as I would like)

    
    
    config = Configuration()
    
    # Inspect the state of the attributes and properties.
    print("\nInitial configuration state:")
    print("config.rootconfig: %s" % config.root_config)
    print("config.group.sub_config: %s" % config.group.sub_config)
    print("config.group.sub_property: %s (calculated)" % config.group.sub_property)
    
    # Inspect whether the properties compute the correct value after we alter
    # some attributes.
    config.root_config = 4
    config.group.sub_config = 5
    
    print("\nState after modifications:")
    print("config.rootconfig: %s" % config.root_config)
    print("config.group.sub_config: %s" % config.group.sub_config)
    print("config.group.sub_property: %s (calculated)" % config.group.sub_property)
    

**The behavior:** (output of execution of all of the above code, as expected)

    
    
    Initial configuration state:
    config.rootconfig: 2
    config.group.sub_config: 3
    config.group.sub_property: 6 (calculated)
    
    State after modifications:
    config.rootconfig: 4
    config.group.sub_config: 5
    config.group.sub_property: 20 (calculated)
    

**Why I don't like it:**

Storing configuration data in class definitions inside of the main object's
`__init__()` doesn't feel elegant. Especially having to instantiate them
immediately after definition like that. Ugh. I can deal with that for the
parent class, sure, but doing it in a constructor...

Storing the same classes outside the main `Configuration` object doesn't feel
elegant either, since properties in the inner classes may depend on the
attributes of `Configuration` (or their siblings inside it).

I could deal with defining the functions outside of everything, so inside
having things like

    
    
    @property
    def solar_zenith_angle(self):
       return calculate_zenith(self.solar_azimuth_angle)
    

but I can't figure out how to do something like

    
    
    @property
    def solar.zenith_angle(self):
        return calculate_zenith(self.solar.azimuth_angle)
    

(when I try to be clever about it I always run into `<property object at
0xXXXXX>`)

So what is the right way to go about this? Am I missing something basic or
taking a very wrong approach? Does anyone know a clever solution?

Help! My python code isn't beautiful! I must be doing something wrong!


Answer: Phil,

Your hesitation about func-y config is very familiar to me :)

I suggest you to store your config not as a python file but as a structured
data file. I personally prefer YAML because it looks clean, just as you
designed in the very beginning. Of course, you will need to provide formulas
for the auto calculated properties, but it is not too bad unless you put too
much code. Here is my implementation using PyYAML lib.

The config file (config.yml):

    
    
    day_of_year: 138
    time_of_day: 36000 # seconds
    solar:
      azimuth_angle: 73 # degrees
      zenith_angle: !property 90 - self.azimuth_angle
    

The code:

    
    
    import yaml
    
    yaml.add_constructor("tag:yaml.org,2002:map", lambda loader, node:
        type("Config", (object,), loader.construct_mapping(node))())
    
    yaml.add_constructor("!property", lambda loader, node:
        property(eval("lambda self: " + loader.construct_scalar(node))))
    
    config = yaml.load(open("config.yml"))
    
    print "LOADED config.yml"
    print "config.day_of_year:", config.day_of_year
    print "config.time_of_day:", config.time_of_day
    print "config.solar.azimuth_angle:", config.solar.azimuth_angle
    print "config.solar.zenith_angle:", config.solar.zenith_angle, "(calculated)"
    print
    
    config.solar.azimuth_angle = 65
    print "CHANGED config.solar.azimuth_angle = 65"
    print "config.solar.zenith_angle:", config.solar.zenith_angle, "(calculated)"
    

The output:

    
    
    LOADED config.yml
    config.day_of_year: 138
    config.time_of_day: 36000
    config.solar.azimuth_angle: 73
    config.solar.zenith_angle: 17 (calculated)
    
    CHANGED config.solar.azimuth_angle = 65
    config.solar.zenith_angle: 25 (calculated)
    

The config can be of any depth and properties can use any subgroup values. Try
this for example:

    
    
    a: 1
    b:
      c: 3
      d: some text
      e: true
      f:
        g: 7.01
    x: !property self.a + self.b.c + self.b.f.g
    

Assuming you already loaded this config:

    
    
    >>> config
    <__main__.Config object at 0xbd0d50>
    >>> config.a
    1
    >>> config.b
    <__main__.Config object at 0xbd3bd0>
    >>> config.b.c
    3
    >>> config.b.d
    'some text'
    >>> config.b.e
    True
    >>> config.b.f
    <__main__.Config object at 0xbd3c90>
    >>> config.b.f.g
    7.01
    >>> config.x
    11.01
    >>> config.b.f.g = 1000
    >>> config.x
    1004
    

**UPDATE**

Let us have a property config.b.x which uses both self, parent and subgroup
attributes in its formula:

    
    
    a: 1
    b:
      x: !property self.parent.a + self.c + self.d.e
      c: 3
      d:
        e: 5
    

Then we just need to add a reference to parent in subgroups:

    
    
    import yaml
    
    def construct_config(loader, node):
        attrs = loader.construct_mapping(node)
        config = type("Config", (object,), attrs)()
        for k, v in attrs.iteritems():
            if v.__class__.__name__ == "Config":
                setattr(v, "parent", config)
        return config
    
    yaml.add_constructor("tag:yaml.org,2002:map", construct_config)
    
    yaml.add_constructor("!property", lambda loader, node:
        property(eval("lambda self: " + loader.construct_scalar(node))))
    
    config = yaml.load(open("config.yml"))
    

And let's see how it works:

    
    
    >>> config.a
    1
    >>> config.b.c
    3
    >>> config.b.d.e
    5
    >>> config.b.parent == config
    True
    >>> config.b.d.parent == config.b
    True
    >>> config.b.x
    9
    >>> config.a = 1000
    >>> config.b.x
    1008
    

