Faster way to concat dataframes with different columns vertically I am trying to concat two huge dataframes (around 15GB) like the following:

a_df:

            A       B       C       D
2018-10-16  a16     b16     c16     d16
2018-10-17  a17     b17     c17     d17
2018-10-18  a18     b18     c18     d18
2018-10-19  a19     b19     c19     d19
2018-10-20  a20     b20     c20     d20


b_df:

            A       C       Y       Z       
2018-10-21  a21     c21     y21     z21     
2018-10-22  a22     c22     y22     z22     


to get a final dataframe:

            A       B       C       D       Y       Z
2018-10-16  a16     b16     c16     d16
2018-10-17  a17     b17     c17     d17
2018-10-18  a18     b18     c18     d18
2018-10-19  a19     b19     c19     d19
2018-10-20  a20     b20     c20     d20
2018-10-21  a21             c21             y21     z21
2018-10-22  a22             c22             y22     z22


I am trying that code:

dfs = [a_df, b_df]

unique_columns = []
for df in dfs:
    for column in df.columns:
        unique_columns.append(column)
unique_columns = sorted(set(unique_columns))

same_columns_dfs = []
for df in dfs:
    empty_df = pd.DataFrame(np.nan, index=df.index, columns=unique_columns)
    empty_df.loc[:, df.columns] = df.values
    same_columns_dfs.append(empty_df)

final_df = pd.concat(same_columns_dfs, axis=0, sort=True)


Is there a more efficient way to do this?
Yep, you can do this in one line, no need to create other lists or empty dataframes when concat will cover your use-case entirely:

import pandas as pd
# to artificially fill a dataframe
import random

# for the purpose of transparency, here's how the dataframes were constructed
a_cols = list('ABCD')
b_cols = list('ACYZ')
a_index = [f'2018-10-{i}' for i in range(16, 21)]
b_index = [f'2018-10-{i}' for i in range(21, 23)]

# creation of the two dataframes
a = pd.DataFrame([{col: random.randint(0, 10) for col in a_cols} for i in a_index], index=a_index)
            A  B  C  D
2018-10-16  6  8  7  4
2018-10-17  5  6  2  5
2018-10-18  4  1  6  5
2018-10-19  4  3  7  9
2018-10-20  7  6  7  5

b = pd.DataFrame([{col: random.randint(0, 10) for col in b_cols} for i in b_index], index=b_index)
            A  C  Y   Z
2018-10-21  1  9  4  10
2018-10-22  1  6  4   6


# and here's your function call
pd.concat([a, b], axis=0, sort=True)
            A    B  C    D    Y     Z
2018-10-16  6  8.0  7  4.0  NaN   NaN
2018-10-17  5  6.0  2  5.0  NaN   NaN
2018-10-18  4  1.0  6  5.0  NaN   NaN
2018-10-19  4  3.0  7  9.0  NaN   NaN
2018-10-20  7  6.0  7  5.0  NaN   NaN
2018-10-21  1  NaN  9  NaN  4.0  10.0
2018-10-22  1  NaN  6  NaN  4.0   6.0


Now the only issue would be the RAM you have at your disposal, and since that wasn't mentioned, I'm assuming this is sufficient. If you are looking for a more memory-optimized solution, we would have to see how those frames are created, since pandas will always load the entire structure into RAM