I am trying to iteratively add some pandas dataframe that I read from a set of csv files, and after the 16th file or so I get a memory error. The new files are pandas of around 300k rows.

Is there a way to do this in the hard drive (for example with hdf5) or in a more memory efficient way?

See code below. Note that sum\_of\_all\_files start as an empty dataframe.

```
sum_of_all_files = pd.DataFrame()
for file_name in list_of_files:
    file_df=pd.read_csv(file_name,index_col=0,header=None).dropna()
    sum_of_all_files=sum_of_all_files.add(file_df,fill_value=0, axis='index')

```

Thanks!

**EDIT**: I want to add by index, i.e. if two rows have the same index, add them. I have corrected the code above by adding " axis='index' " in the last line.