Estimating Dataframe memory usage from file sizes
You could open each CSV, read first 1000 lines only into DataFrame, and then check memory usage. Then scale estimated memory usage by number of lines in the file.
Note that memory_usage() isn't accurate with default arguments, because it won't count strings' memory usage. You need memory_usage(deep=True), although that might overestimate memory usage in some cases. But better to overestimate than underestimate.
If I have a list of files in a directory is it possible to estimate a memory use number that would be taken up by reading or concatenating the files using pd.read_csv(file) or pd.concat([df1, df2])?
I would like to break these files up into concatenation 'batches' where each batch will not exceed a certain memory usage so I do not run into local memory errors.
Using os.path.getsize() will allow me to obtain the file sizes  and df.memory_usage() will tell me how much memory the dataframe will use once it's already read in but is there a way to estimate this with just the files themselves?