Any one read the over 100gb csv file and successfully concatenation?
Its too big file to handle by standard way. You could do it by chunk
for chunk in reader:
    chunk['col1']=chunk['col1']**2 #and so on

Or dump yours csv file to database.
number of rows
num=0 
for chunk in reader: 
    num+=1 
num_of_rows = num*lines_of_chunk

#work around in bash and python
import subprocess
subprocess.check_output([&quot;wc&quot;,&quot;-l&quot;, &quot;file.csv&quot;])