How to persist a python dictionary?
I would recommend persisting as json using pandas and reloading again as needed with pandas. Pandas will make the reading and writing really easy for you. This allows you to have the superset of columns in the dataframe with nulls in the spots that are missing data. 
This saves you from needing to do a key value pair storage option as well. 
Plus, it looks like your data is already in a format close to json. Once you have it loaded back into the dataframe, querying as you need to will also be simple.
I have a python program that takes in a list of objects of different types, and for each type, the program will output a dictionary of key/value attributes where the key is some property of the given object's type, and value is its computed result.
To make it more concrete, my program takes in a list of 2000 objects, of 3 unique types: Cars, Planes, Ships. And for a car, it produces: 

{"ID": , "Horsepower":120.5, "Fuel Efficiency": 19,
  "Turning Radius":20, "Weight":500}

For ship, it's 

{"ID": , "Displacement": 1000.5, "Fuel Efficiency": 8,
  "Weight": 2000}

For plane, it's 

{"ID": , "Engine Size": 200.5, "Fuel Efficiency": 8,
  "Weight": 2000}

So you can see, for each type, the number and content of its dictionary output is different, while they may all share some common fields such as "ID" (unique across different objects), "Weight", etc.
And tomorrow there could be a new type that needs to be supported by the program with a similar output structure.
The question is what is the best way to persist these outputs, for easy querying/aggregation later on. Such as: give me all planes with a weight >= 1000, or give me all weights of all cars whose horse power is between 200 and 300.
Lets say we use pandas dataframe as our storage format, I am faced with 2 choices:
Take a union of all keys of all product types, and create a pandas df with those keys as column, and each row represents each product's output, and it may have None in a given column depending on product. This essentially creates a sparse matrix. And the column names can grow because new product types can have new keys as outputs.
Create a pandas df with 3 columns: ID, Key,  Value.
Which one do you recommend or is there an obvious third option I'm missing?