Faster way to concat dataframes with different columns vertically
Yep, you can do this in one line, no need to create other lists or empty dataframes when concat will cover your use-case entirely:

import pandas as pd
# to artificially fill a dataframe
import random

# for the purpose of transparency, here's how the dataframes were constructed
a_cols = list('ABCD')
b_cols = list('ACYZ')
a_index = [f'2018-10-{i}' for i in range(16, 21)]
b_index = [f'2018-10-{i}' for i in range(21, 23)]

# creation of the two dataframes
a = pd.DataFrame([{col: random.randint(0, 10) for col in a_cols} for i in a_index], index=a_index)
            A  B  C  D
2018-10-16  6  8  7  4
2018-10-17  5  6  2  5
2018-10-18  4  1  6  5
2018-10-19  4  3  7  9
2018-10-20  7  6  7  5

b = pd.DataFrame([{col: random.randint(0, 10) for col in b_cols} for i in b_index], index=b_index)
            A  C  Y   Z
2018-10-21  1  9  4  10
2018-10-22  1  6  4   6


# and here's your function call
pd.concat([a, b], axis=0, sort=True)
            A    B  C    D    Y     Z
2018-10-16  6  8.0  7  4.0  NaN   NaN
2018-10-17  5  6.0  2  5.0  NaN   NaN
2018-10-18  4  1.0  6  5.0  NaN   NaN
2018-10-19  4  3.0  7  9.0  NaN   NaN
2018-10-20  7  6.0  7  5.0  NaN   NaN
2018-10-21  1  NaN  9  NaN  4.0  10.0
2018-10-22  1  NaN  6  NaN  4.0   6.0


Now the only issue would be the RAM you have at your disposal, and since that wasn't mentioned, I'm assuming this is sufficient. If you are looking for a more memory-optimized solution, we would have to see how those frames are created, since pandas will always load the entire structure into RAM