how to work with a large dataset in python? for a training purposes, in order to start machine learning and data analysis with python I work on a pretty small dataset (20000 rows, 600MB) .But since couple days I decided to go further so  I have download a large data set of 1Go.

I wanted to do some analysis and apply machine learning on it so I tried to read the csv file with pyhton in jupiter nootebook and th file is still loading after more than 3 hours.

So I will would like to know  what are the best practices/process to follow when you have to work with large dataset ?
Before exploring more sophisticated tools like Spark or Dask, one option would be to read the data in chunks instead of loading the whole file. For example, if you are using pandas the read_csv method accepts chunksize argument.

The main idea is that often what you need to do is reduce each chunk down to something much smaller with only the parts you need (some average/count/sum etc.. of some classes) and store a series/array/dict etc accordingly.