How to handle a huge stream of JSON dictionaries?
Question: I have a file that contains a stream of JSON dictionaries like this:

    
    
    {"menu": "a"}{"c": []}{"d": [3, 2]}{"e": "}"}
    

It also includes nested dictionaries and it looks like I cannot rely on a
newline being a separator. I need a parser that could be used like this:

    
    
    for d in getobjects(f):
      handle_dict(d)
    

The point is that it would be perfect if the iteration only happened at the
root level. Is there a Python parser that would handle all JSON's quirks? I am
interested in a solution that would work on files that wouldn't fit into RAM.


Answer: I think
[JSONDecoder.raw_decode](https://docs.python.org/2/library/json.html#json.JSONDecoder.raw_decode)
may be what you're looking for. You may have to do some string formatting to
get it in the perfect format depending on newlines and such, but with a bit of
work, you'll probably be able to get something working. See this example.

    
    
    import json
    jstring = '{"menu": "a"}{"c": []}{"d": [3, 2]}{"e": "}"}'
    substr = jstring
    decoder = json.JSONDecoder()
    
    while len(substr) > 0:
        data,index = decoder.raw_decode(substr)
        print data
        substr = substr[index:]
    

Gives output:

    
    
    {u'menu': u'a'}
    {u'c': []}
    {u'd': [3, 2]}
    {u'e': u'}'}
    

