Combining nested collection in mongodb with documents written from parallel nodes
Question: I'm in the process of figuring out whether I can use MongoDB to help with our
storage and processing issues. The idea is that computation will be done on
each node in a multi-processed way and written to mongodb with a unique
mongodb **ObjectId**. The data structure is as follows below from a
dictionary:

    
    
    {a: {b: {c: [100, 200, 300]} }
    

a, b and c are integer keys

When the computation has been done and all the records written out to mongo,
the documents must be combined such that the we group by the top level a, then
by b and then c. So two documents might contain (**Example A**):

document1: `{24: {67: {12: [100, 200]}}}`

document2: `{24: {68: {12: [100, 200]}}}`

_then if we combine:_

combined: `{24: {67: {12: [100, 200]}, 68: [100, 200]}}`

and if we had another couple of documents (**ExampleB**):

document1: `{24: {67: {12: [100, 200]}}}`

document2: `{24: {67: {12: [300, 400]}}}`

combined: `{24: {67: {12: [100, 200, 300, 400]}}}`

What is the best approach to combine these nested structures. I could loop
through each document by hand and do this say in python, but is there are
smarter approach to this? I need to keep the underlying data structure.


Answer: What is not smart in doing the aggregation with python? Consider the following
function:

    
    
    def aggregate(documents, base_document=None, unique=True):
        # use unique=False to keep all values in the lists, even if repeated
        # like [100, 100, 200, 300], leave it True otherwise
        for doc in documents:
            if isinstance(doc, list):
                if base_document is None: base_document = []
                for d in doc:
                    base_document.append(d)
                if unique==True: base_document = set(base_document)
                base_document = sorted(base_document)
            else:
                if base_document is None: base_document = {}
                for d in doc:
                    b = base_document[d] if d in base_document \
                        else [] if isinstance(doc[d], list) else {}
                    base_document[d] = aggregate([doc[d]], base_document=b)
        return base_document
    

Tested with the following set of documents, it produces the aggregation :

    
    
    documents = [   {20: {55: { 7: [100, 200]}}},
                    {20: {68: {12: [100, 200]}}},
                    {20: {68: {12: [500, 200]}}},
                    {23: {67: {12: [100, 200]}}},
                    {23: {68: {12: [100, 200]}}},
                    {24: {67: {12: [300, 400]}}},
                    {24: {67: {12: [100, 200]}}},
                    {24: {67: {12: [100, 200]}}},
                    {24: {67: {12: [300, 500]}}},
                    {24: {67: {13: [600, 400]}}},
                    {24: {67: {13: [700, 900]}}},
                    {24: {68: {12: [100, 200]}}},
                    {25: {67: {12: [100, 200]}}},
                    {25: {67: {12: [300, 400]}}},   ]
    
    from pprint import pprint
    pprint(aggregate(documents))
    
    ''' 
    {20: {55: {7: [100, 200]}, 68: {12: [100, 200, 500]}},
     23: {67: {12: [100, 200]}, 68: {12: [100, 200]}},
     24: {67: {12: [100, 200, 300, 400, 500], 13: [400, 600, 700, 900]},
          68: {12: [100, 200]}},
     25: {67: {12: [100, 200, 300, 400]}}}
    '''
    

