Reading large CSV file with Pandas freezes computer
You're probably loading all of the data in your RAM, thus allocating all memory available, forcing your system to rely on swap memory (writing temporary data to the disk, which is MUCH slower).
It should solve the issue if you split the data into chunks that fit in your memory. Maybe 1 GB each?
I am working with a relatively large CSV file in Python. I am using the pandas read_csv function to import it. The data is on a shared folder at work and around 25 GB.
I have 2x8 GB RAM and an Intel Core i5 processor and using the juypter notebook. While loading the file the RAM Monitoring goes up to 100%. It stays at 100% or 96% for some minutes and then my computer clock stopped and my screen is frozen. Even if I wait 2 hours my computer is not able to use any more, so I have to restart.
My question is:
Do I need to split the data? Would it help? Or is it a general performance problem with my laptop?
It is the first time that I am working with such a 'large' dataset (I still think 25 GB is not too much.)