Memory efficient way to merge multiple Pandas DataFrames I am trying to merge multiple pandas data-frames that I read from a file.
Here is the function I wrote:

def merge_files(args):

    list_df = {}
    with args.input as data_in:
        for line in data_in:
            line = line.strip()
            df = pd.read_csv(line, sep="\s+")
            df.drop_duplicates(inplace = True)
            # change second column name (from the file)
            df.columns = ["sample", line.split("/")[-4]]
            list_df[line.split("/")[-4]] = df
    # Merging all the files into one and fill the missing ratio values with 0
    df_merged = reduce(lambda  left,right: pd.merge(left,right,on=['sample'], how='outer'), list(list_df.values())).fillna(0)
    df_merged.to_csv(args.output, sep='\t', encoding='utf-8', index=False)


But this approach consumes too much memory, Is there a better solution?

Update  


The function is called in a script and given args contains input file
(contain locations of the csv files one per line) and output file.
The input file contains locations of 20 files each is ~2M in size and
contains ~59133 lines.
Using Python 3, Pandas version '0.25.2'
The issue was resolved, there was a duplication in keys in more than two DF, which was leading to memory usage increase.
Thanks.