Which tool should I use for combining this large dataset?
I would suggest trying to do it in batches. The underlying issue could well still be memory related in some way, as the merge method makes copies of its input and so is not memory efficient at all.

As an example, you could read in 10 files, create the desired output, as you have done already. Repeat this for files 10-20, then 20-30 then 30-40. The finally four the four files that you have created. It is a bit of an annoyance, but sometimes these little workarounds get the job done.



[EDIT]

Another option might be to use the more involved memory management during reading, option via the chunksize argument of pd.read_csv(). This will read parts of the file into memory in chunks, as the name suggests. If you do this in a loop, it should put an upper limit on the memory usage. For example (untested):

chunksize = 50e6        # 50 Mb

for single_file in list_of_file_paths:
    for i, chunk in enumerate(pd.read_csv(single_file, chunksize=chunksize):
        if i == 0:
            result = chunk
        else:
            result = pd.merge((result, chunk), copy=False, how= ...)


You may need to do something with the indivdual chunks before merging them.

Additionally, note that I set the copy argument in merge to False, which might help - the documentation is a little vague as to how it saves memory.