How to deal with memory insufficient read by pandas in python I use pandas.read_csv to read a huge file for machine learning, but I got memory error.

Someone recommend me to set arg chunksize but I need sort, random access...etc. So I need to load whole data into memory or use another way.

Some ways I think it's possible is Hadoop. Another one is incremental training, but it's like reading chunksize in read_csv

Or other software/library/ways can I use?
I would suggest you to use Dask. I used it successfully when I had to read large data with my 4GB RAM. You can get more details here.

To read a CSV, you can do the following:

import dask.dataframe as dd

csv_file = 'data.csv'
df = dd.read_csv(csv_file)