saving multiple dicts and variables in disk
Question: My code typically runs for several days and spews out intermediate results as
Python dicts and float variables. I cannot afford to append all intermediate
values to a Python list and save all outputs to a physical file using pickle
etc in one go after the run. This is because, in case there is interruption, I
might loose all data I have collected so far.

So I have to dump (and append) multiple dicts and other variables
periodically. What is the best way to do this ? I've looked at JSON (but not
sure how to do store (and later read) multiple JSON objects from a single
file. Can I append my data to a pickle file ? Saving these data in a simple
text file or csv file would be my last resort.


Answer: Use [`json`](https://docs.python.org/2/library/json.html). This is a good
variant for conservation with the ability to edit the file:

    
    
    import json
    
    list_of_dict = [
        {'a': 1},
        {'b': 2}
    ]
    
    with open('filename.json', 'w') as f:
        json.dump(list_of_dict, f, sort_keys=True, indent=4)
    

