one of the easiest ways to store data efficiently in binary format is using python' builtin pickle serialization convenientlypandas objects all have save method which writes the data to disk as picklein [ ]frame pd read_csv('ch /ex csv'in [ ]frame out[ ] message hello world foo in [ ]frame save('ch /frame_pickle'you read the data back into python with pandas loadanother pickle convenience functionin [ ]pd load('ch /frame_pickle'out[ ] message hello world foo pickle is only recommended as short-term storage format the problem is that it is hard to guarantee that the format will be stable over timean object pickled today may not unpickle with later version of library have made every effort to ensure that this does not occur with pandasbut at some point in the future it may be necessary to "breakthe pickle format using hdf format there are number of tools that facilitate efficiently reading and writing large amounts of scientific data in binary format on disk popular industry-grade library for this is hdf which is library with interfaces in many other languages like javapythonand matlab the "hdfin hdf stands for hierarchical data format each hdf file contains an internal file system-like node structure enabling you to store multiple datasets and supporting metadata compared with simpler formatshdf supports on-the-fly compression with variety of compressorsenabling data with repeated patterns to be stored more efficiently for very large datasets that don' fit into memoryhdf is good choice as you can efficiently read and write small sections of much larger arrays there are not one but two interfaces to the hdf library in pythonpytables and pyeach of which takes different approach to the problem py provides directbut high-level interface to the hdf apiwhile pytables abstracts many of the details of binary data formats www it-ebooks info