Extracting contents of dictionary contained in Pandas dataframe to make new dataframe columns I created a Pandas dataframe from a MongoDB query.

c = db.runs.find().limit(limit)
df = pd.DataFrame(list(c))


Right now one column of the dataframe corresponds to a document nested within the original MongoDB document, now typed as a dictionary. The dictionary is in the run_info column. 

I would like to extract some of the dictionary's values to make new columns of the data frame. Is there a general way to do this? If not, what is the best brute force method specific to my data?

Here is my brute force approach, and it's horrendous. Even for a dataframe of only 200 rows, this is taking several minutes:

run_info = df['run_info']
# want to extra values from dictionary stored in column df['run_info']
# some values are within dictionaries nested within top dictionary
# these are 'author' and 'weather'
for i in range(len(df['run'])):
    g = run_info[i]
    if g.get('weather'):
        for name in weather_list:
            if g.get('weather').get(name):
                try:
                    df.set_value(i, name, g['weather'][name])
                except:
                    pass
    if g.get('author'):
        for name in author_list:
            if g.get('author').get(name):
                try:
                    df.set_value(i, name, g['author'][name])
                except:
                    pass
    for name in name_list:
        if g.get(name):
            try:
                if name is 'local_start_time':
                    df.set_value(i, name, g[name][0:19])
                else:
                    df.set_value(i, name, g[name])
            except:
                pass


I'd appreciate any and all suggestions for how to improve the speed here. Also I am using the try...except because once in a while the encoding on some funky character is causing a ValueError.
I suspect the try-except in the loops are the main cause of the slowness. It's not clear at which point you get ValueError due to "funky" data. It would be good to clean that up, get rid of the try-catch, and I think your solution will get noticeably faster.

Another small thing, that might not make a noticeable difference at all, you have some repeated lookups, like in this snippet:

if g.get('weather'):
    for name in weather_list:
        if g.get('weather').get(name):


Here, the g.get('weather') bit is executed twice. It would be better to save the result of the first call, and then reuse that. Apply this logic to other similar places. Although it might not make a practical difference, avoiding duplicated logic is a good practice.