How to reversibly store and load a Pandas dataframe to/from disk
Another quite fresh test with to_pickle().
I have 25 .csv files in total to process and the final dataframe consists of roughly 2M items.
(Note: Besides loading the .csv files, I also manipulate some data and extend the data frame by new columns.)
Going through all 25 .csv files and create the dataframe takes around 14 sec.
Loading the whole dataframe from a pkl file takes less than 1 sec
Right now I'm importing a fairly large CSV as a dataframe every time I run the script. Is there a good solution for keeping that dataframe constantly available in between runs so I don't have to spend all that time waiting for the script to run?