Merging dataframes in Pandas is taking a surprisingly long time
There are a couple of points I can maybe give to help.

Firstly, Pandas is not great at merging multiple large dataframes in general because every time you merge a new dataframe to an old one, it makes a copy of both to make a third dataframe - this obviously starts taking a lot of time as your master dataframe grows in each step.

Secondly (untested), you could try to overcome any memory bottlenecks in the merging steps by trying something dumb, like merging two or three files, then writing it to disk. Repeat that until you have 30 larger CSV files, then do it again so you have, say 10 files, and so on. I am not sure at which point the python garbage collection would kick in with you single reduce function, so it could even be the case that you start using SWAP memory, which would make things really slow! This solution could at least prevent that from happening. You should perform each repeat in a new Python session.

One other idea, as you are reading the data from CSV files: you could use a terminal tool to simply paste all the CSV files into one file, then read that into pandas once. Then perform required clean up actions. The cleaning might be a pain (given your merge on requirements, but you would likely overcome the long waiting times for pandas to make the copies. If you files are all in one folder, you can run the following in terminal to put them all in one file

cat *.csv &gt; merged.csv


There are other methods here, in case you need to leave out the header line etc.

Lastly I would recommend having a look at the data.table package, which is now functional in Python - it started in the R world. It has a much more robust memory management system (built on ideas from databases) and is able to store huge amounts of data in a single "DataTable", in cases where Pandas would crash. It is a great package to fill the gap between Pandas and Dask (both awesome packages!).

Here is a video, showing how Python data.table works and also shows some benchmarks.