"join" two list of dictionaries on Id I have two lists of dictionaries where both have the same &quot;id&quot; key but otherwise different data.
list1 = [{&quot;id&quot;:1, &quot;spam&quot;: 3, &quot;eggs&quot;: 5}, {&quot;id&quot;: 2, &quot;spam&quot;: 5, &quot;eggs&quot;: 7}]
list2 = [{&quot;id&quot;:1, &quot;knights&quot;: 5, &quot;coconuts&quot;: 2}, {&quot;id&quot;: 2, &quot;knights&quot;: 3, &quot;cocounts&quot;: 8}]

I know I can do a nested loop something like this (I know I can use dict.update as well, but just writing it out as an exampel):
for id in list1:
    for i in list2:
        if id['id'] == i['id]:
           id['knights'] = i['knights']
           id['coconuts'] = i['coconuts']

but can anyone tell me the more pythonic or quicker way to do this? matching two lists with millions of rows in each does not practically work this way.
Expected result is:
[{&quot;id&quot;: 1, &quot;spam&quot;: 3, &quot;eggs&quot;: 5, &quot;knights&quot;: 5, &quot;coconuts&quot;: 2}, {&quot;id&quot;: 2, &quot;spam&quot;: 5, &quot;eggs&quot;: 7, &quot;knights&quot;: 3, &quot;coconuts&quot;: 8}]
To improve time complexity you must sometimes trade-off space complexity. It's known as space-time tradeoff
You can transform list1 into a dictionary where key is id and value is the dictionary itself.
list1 = {d['id']: d for d in list1}
# print(list1)
# {1: {'id': 1, 'spam': 3, 'eggs': 5, 'knights': 5, 'coconuts': 2},
#  2: {'id': 2, 'spam': 5, 'eggs': 7, 'knights': 3, 'coconuts': 8}}

Now, iterate through list2 and update list1 based on id
for d in list2:
    list1[d['id']].update(d)

#print(list1)
# {1: {'id': 1, 'spam': 3, 'eggs': 5, 'knights': 5, 'coconuts': 2},
#  2: {'id': 2, 'spam': 5, 'eggs': 7, 'knights': 3, 'coconuts': 8}}

Take dict.values() to get desired output
print(list(list1.values()))
# [{'id': 1, 'spam': 3, 'eggs': 5, 'knights': 5, 'coconuts': 2},
#  {'id': 2, 'spam': 5, 'eggs': 7, 'knights': 3, 'coconuts': 8}]

Dealing with Edge cases
The above approach fails when list2 has an id which doesn't exist in list1. If that's the case then use this and you want to add that new id to output too. Using dict.setdefault
# for example list2 is as below
# [{'id': 1, 'knights': 5, 'coconuts': 2},
#  {'id': 2, 'knights': 3, 'coconuts': 8},
#  {'id': 3, 'knight': 4, 'coconuts': 5}] -&gt; new `id` that's not in `list1`

# Here we leverage on `dict.setdefault`

for d in list2:
    list1.setdefault(d['id'], dict()).update(d)

print(list1)
# {1: {'id': 1, 'spam': 3, 'eggs': 5, 'knights': 5, 'coconuts': 2},
#  2: {'id': 2, 'spam': 5, 'eggs': 7, 'knights': 3, 'coconuts': 8},
#  3: {'id': 3, 'knight': 4, 'coconuts': 5}}

# To get list of dicts use `list(list1.values())

In case you dont want to update with new id, we can resolve this using if block here.
for d in list2:
    if d['id'] in list1:
        list1[d['id']].update(d)
# {1: {'id': 1, 'spam': 3, 'eggs': 5, 'knights': 5, 'coconuts': 2},
#  2: {'id': 2, 'spam': 5, 'eggs': 7, 'knights': 3, 'coconuts': 8}}
# No `id` 3 present in output

Using libraries that support vectorization?
If you are working on millions of rows you can leverage on pandas which has many vectorized operations. In your case we can use DataFrame.merge
import pandas as pd

df1 = pd.DataFrame(list1)
df2 = pd.DataFrame(list2)

# Now, you have to merge them on `id`

out = df1.merge(df2) 

#    id  spam  eggs  knights  coconuts
# 0   1     3     5        5         2
# 1   2     5     7        3         8

To get output as a list of dict you df.to_dict with orient param set to records
out.to_dict(orient='records')

# [{'id': 1, 'spam': 3, 'eggs': 5, 'knights': 5, 'coconuts': 2},
#  {'id': 2, 'spam': 5, 'eggs': 7, 'knights': 3, 'coconuts': 8}]

If you want include new ids then set how param of df.merge to 'outer' by default it's 'inner'
df1.merge(df2, how='outer')

#    id  spam  eggs  knights  coconuts
# 0   1   3.0   5.0        5         2
# 1   2   5.0   7.0        3         8
# 2   3   NaN   NaN        4         5


Code Review

Naming variables
list1 = [...]
list2 = [...]

A new reader might not understand what's list1 and list2. Name them so they make sense to everyone.
current_users_details = [...]
new_details = [...]

This is just an example, name them accordingly. What I do is ask myself what is it storing in the data-structure?... family member details? then variable name is family_members_details, user details then user_details, pi value? then PI_VALUE... etc.

Violation of E231
list1 = [{&quot;id&quot;:1, &quot;spam&quot;: 3, &quot;eggs&quot;: 5}, {&quot;id&quot;: 2, &quot;spam&quot;: 5, &quot;eggs&quot;: 7}]
list2 = [{&quot;id&quot;:1, &quot;knights&quot;: 5, &quot;coconuts&quot;: 2}, {&quot;id&quot;: 2, &quot;knights&quot;: 3, &quot;cocounts&quot;: 8}]

The above lines use inconsistent use of spaces after :, PEP-8 suggest add a space after :.

Violation of E501
list2 = [{&quot;id&quot;:1, &quot;knights&quot;: 5, &quot;coconuts&quot;: 2}, {&quot;id&quot;: 2, &quot;knights&quot;: 3, &quot;cocounts&quot;: 8}]

Lines longer than 79 characters, PEP-8 standard suggests to keep characters in a line &lt;= 79

You have missing ' here if id['id'] == i['id]:

Reusability
Make a function which takes both the lists and updates them.
def update_details(current_details, to_update_with):
    &quot;&quot;&quot;
    updates user details
    &quot;&quot;&quot;

    details = {d['id']: d for d in current_details}
    for user in to_update_with:
        details[user['id']].update(user)
    return details

Write a much better docstring :P



I suggest using black python code formatter.
ch3ster@ch3ster:~$ black --line-length 79 your_file.py

Formatted code
Formatted your code using black.
list1 = [{&quot;id&quot;: 1, &quot;spam&quot;: 3, &quot;eggs&quot;: 5}, {&quot;id&quot;: 2, &quot;spam&quot;: 5, &quot;eggs&quot;: 7}]
list2 = [
    {&quot;id&quot;: 1, &quot;knights&quot;: 5, &quot;coconuts&quot;: 2},
    {&quot;id&quot;: 2, &quot;knights&quot;: 3, &quot;cocounts&quot;: 8},
]

for d in list2:
    list1[d[&quot;id&quot;]].update(d)

print(list(list1.values())) # Equivalent to print([*list1.values()])