Read list of dictionaries with nested dictionaries
Currently you iterate over each dictionary twice. You can do it in one pass:

for v in d:
    commodities_reference = []
    for k, g in v.items():
        if isinstance(g, dict):
            commodities_reference.append(g["id"])
            commodities_reference.append(g["name"])
        else:
            commodities_reference.append(g)


Note that this appends the values when it encounters it. This means that in Python &lt; 3.7 (cPython &lt; 3.6) it is not guaranteed that the dictionary is actually the last item to be looked at, since dictionaries were not to guaranteed to be in insertion order.

You could even make this a generator and slightly more general:

def get_values_recursive(x):
    for value in x.values():
        if isinstance(value, dict):
            yield from get_values_recursive(value)
        else:
            yield value

for v in d:
    commodities_reference = list(get_values_recursive(v))
    # do something with it...
    print(commodities_reference)


When using the given example, this is the result:

&gt;&gt;&gt; list(get_values_recursive(v))
[1, 'Explosives', 1, 294, 0, 755, 1774, 99, 18, 176, 924, 0, 128049204, 1, 'Chemicals']


When putting your code into a function, this generator is almost twice as fast with the given v:

In [13]: %timeit op(v)
5.32 µs ± 43.3 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

In [14]: %timeit list(get_values_recursive(v))
3.64 µs ± 10.8 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)


Note that both take on the order of micro seconds, so unless you need to process more than 100000 items per second your bottleneck is probably in those SQL statements and how you execute them.