Caching Pandas Dataframe by Serialization or In-memory KV Store
I have a DF of ~ 1 GB of plain text data. Assuming the dumping to disk is always slower than reading I compared HDF5 write performance with pickle.
HDF5 took 35 sec while pickle did 190 sec. So, you could consider using HDF5 instead of pickle
Which method of caching pandas DataFrame objcts will provide the highest performance? By storing it to a flat file on disk using pickle, or by storing it in a key-value store like Redis?