title: Fastest way to store large files in Python
body: <p>I <a href="https://stackoverflow.com/questions/7626848/maximum-python-object-which-can-be-passed-to-write">recently asked a question</a> regarding how to save large python objects to file. I had previously run into problems converting massive Python dictionaries into string and writing them to file via <code>write()</code>. Now I am using pickle. Although it works, the files are incredibly large (> 5 GB). I have little experience in the field of such large files. I wanted to know if it would be faster, or even possible, to zip this pickle file prior to storing it to memory. </p>