Is Panda appropriate for joining 120 large txt files?
Saying you want to concat in pandas implies that you just want to merge all 150 files together into one file? If so you can iterate through all the files in a directory and read them in as lists of tuples or something like that and just combine them all into one list. Lists and tuples are magnitudes less memory than dataframes, but you won't be able to perform calculations and stuff unless you throw them in as a numpy array or dataframe.
At a certain point, when there is too much data it is appropriate to shift from pandas to spark since spark can use the power and memory from a cluster instead of being restricted to your local machine or servers resources.
I have 120 txt files, all are around 150mb in size and have thousands of columns.  Overall theres definitely more than 1million columns.
When I try to concatenate using pandas I get this error: " Unable to allocate 36.4 MiB for an array with shape (57, 83626) and data type object"... I've tried Jupyter notebook and Spyder, neither work
How can I join the data? Or is this data not suitable for Pandas.
Thanks!