Pandas .to_csv takes more space than memory usage
The easiest way to reduce the size of the csv is to compress it when writing, using the compression parameter in to_csv. For example df.to_csv(compression='gzip').
There are a variety of reasons the memory usage could be so different from the size of the csv on disk, it's a little hard to say without knowing any specifics about the data you're working with.
One generic recommendation is to check the precision of any floating point values in your dataframe, if you're writing a bunch of numbers with 15 decimal points of precision or something that will take up a lot of space. Try truncating these values to the precision you need.
I had a DataFrame whose memory usage was 159.7 MB. When I used .to_csv method to write it in storage the written file was about 400 MB. And when I loaded this file its memory usage was 159.7 MB. Is there an explanation for this difference in sizes and how to write it so that it takes less space in the hard drive ? Thank you for your help