Merge multiple pandas dataframe along with np.where to process values
keyword arguments
I had to check the documentation to find out what the 1 means in df1.drop('new_id', 1). More clear would be df1.drop('new_id', axis=1), or even better: df1.drop(columns=[&quot;type_&quot;, &quot;new_id&quot;])
Don't overwrite raw data
You add columns and values to df1, which makes finding problems harder.
With a bit of reshuffling, and using df.join instead of pd.merge, you can make your intent a bit more clear
df6= df1.join(
    df2.assign(level=&quot;l1&quot;).set_index([&quot;id&quot;, &quot;level&quot;]),
    how=&quot;outer&quot;,
    on=[&quot;id&quot;, &quot;level&quot;],
).join(
    df3.assign(level=&quot;l2&quot;).set_index([&quot;id&quot;, &quot;level&quot;]),
    how=&quot;outer&quot;,
    on=[&quot;id&quot;, &quot;level&quot;],
    lsuffix=&quot;_&quot;,
)

np.where
If you're looking for null, you can use fillna or combine_first. To replace 'l2' with 'l3', you can use Series.replace.
df7 = df6.assign(
    id=df6[&quot;new_id&quot;].fillna(df6[&quot;id&quot;]).astype(int),
    type=df6[&quot;type_&quot;].fillna(df6[&quot;type&quot;]),
    level=df6[&quot;level&quot;].replace(&quot;l2&quot;, &quot;l3&quot;)
).drop(columns=[&quot;type_&quot;, &quot;new_id&quot;])

I don't know whether this approach will be faster, but to me, what happens here is a lot more clear.