MongoDB, Python and PyMongo: Document size too large with BSONObj size is invalid
Question: I am getting this error when writing to Mongo:

    
    
    OperationalFailure caught
    10334
    {u'connectionId': 2365, u'code': 10334, u'ok': 1.0, u'err': u'BSONObj size: 17254820 (0xA4490701) is invalid. Size must be between 0 and 16793600(16MB) First element: 0:
    

This is a normal document full of strings and integers, constructed in Python,
yet it's size seems to be 17,25MB. What would you do?

This is how the data looks like:

    
    
    { date: new Date(1417996800000), 
      visitors: [ { owner: "AS3320 Deutsche Telekom AG", ip: "82.148.15.23", views: 844 }, 
                  { owner: "AS29314 VECTRA S.A.", ip: "173.235.42.25", views: 458 }, 
                    ...
                ]
    }
    

There are many, many elements in that array, yet I am surprised that the
amount surpasses 16MB.

After limiting the size of the array down to 8500 elements, I am getting this
PyMongo error:

    
    
    $ operator made object too large
    


Answer: There are a lot of things to think about when designing your Mongo schemas,
but generally the schema should reflect how you use your data. The [6 Rules of
Thumb for MongoDB Schema
Design](http://blog.mongodb.org/post/87200945828/6-rules-of-thumb-for-mongodb-
schema-design-part-1) article series on the MongoDB Blog is a good start.

My first thought is to turn your documents "inside out" and store this kind of
document in a collection:

    
    
    { date: new Date(1417996800000), owner: "AS3320 Deutsche Telekom AG", ip: "82.148.15.23", views: 844 },
    { date: new Date(1417996800000), owner: "AS29314 VECTRA S.A.", ip: "173.235.42.25", views: 458 },
    ...
    

This way you're not limiting how many unique users you can have per day. If
you are indexing on the `date` field, looking up visitors by date is still
efficient.

If you are not using the `owner` field much, maybe it can be moved to its own
collection as well.

    
    
    { ip: "82.148.15.23", owner: "AS3320 Deutsche Telekom AG"},
    { ip: "173.235.42.25", owner: "AS29314 VECTRA S.A."},
    ...
    

Not a definitive answer, obviously, but it might be a start.

About checking the size of the document before writing (which I don't think is
a good way to get around poor schema design). Data is handled as BSON by
MongoDB internally so you can use the [`bson`
module](https://pypi.python.org/pypi/bson/0.3.3):

    
    
    import bson
    len(bson.dumps(my_document))
    

If you get the "operator made object too large" error, see [this
question](http://stackoverflow.com/questions/11224942/mongodb-update-with-
upsert-fails).

