Efficiently save to disk (heterogeneous) graph of lists, tuples, and NumPy arrays
Question: I am regularly dealing with large amounts of data (order of several GB), which
are stored in memory in NumPy arrays. Often, I will be dealing with nested
lists/tuples of such NumPy arrays. How should I store these to disk? I want to
preserve the list/tuple structure of my data, the data has to be compressed to
conserve disk space, and saving/loading needs to be fast.

(The particular use case I'm facing right now is a 4000-element long list of
2-tuples `x` where `x[0].shape = (201,)` and `x[1].shape = (201,1000)`.)

I have tried several options, but all have downsides:

  * [`pickle`](https://docs.python.org/3.5/library/pickle.html) storage into a `gzip` archive. This works well, and results in acceptable disk space usage, but is extremely slow and consumes a lot of memory while saving.

  * [`numpy.savez_compressed`](http://docs.scipy.org/doc/numpy-1.10.1/reference/generated/numpy.savez_compressed.html). Is much faster than `pickle`, but unfortunately only allows either a sequence of numpy arrays (not nested tuples/lists as I have) or a dictionary-style way of specifying the arguments.

  * Storing into HDF5 through [`h5py`](http://www.h5py.org/). This seems too cumbersome for my relatively simple needs. More importantly, I looked a lot into this, and also there does not seem to be a straightforward way to store heterogeneous (nested) lists.

  * [`hickle`](https://github.com/telegraphic/hickle) seems to do exactly what I want, however unfortunately it's incompatible with Python 3 at the moment (which is what I'm using).

I was thinking of writing a wrapper around `numpy.savez_compressed`, which
would determine the nested structure of the data, store this structure in some
variable `nest_structure`, flatten the full graph, and store both
`nest_structure` and all the flattened data using `numpy.savez_compressed`.
Then, the corresponding wrapper around
[`numpy.load`](http://docs.scipy.org/doc/numpy-1.10.1/reference/generated/numpy.load.html)
would understand the `nest_structure` variable, and re-create the graph and
return it. However, I was hoping there is something like this already out
there.


Answer: You may like the [`shelve`](https://docs.python.org/3/library/shelve.html
"shelve") package. It effectively wraps heterogeneous pickled objects in a
convenient file. `shelve` is oriented more toward a "persistent storage" than
classic save-to-file model.

The main benefit of using `shelve` is that you can conveniently save most
kinds of structured data. The main disadvantage of using `shelve` is that it
is Python-specific. Unlike HDF-5 or saved Matlab files or even simple CSV
files, it isn't so easy to use other tools with your data.

Example of saving (Out of habit, I created objects and copy them to `df`, but
you don't need to do this. You could just save directly to items in `df`):

    
    
    import shelve
    import numpy as np
    a = np.arange(0, 1000, 12)
    b = "This is a string"
    class C(object):
       alpha = 1.0
       beta = [3, 4]
    c = C()
    class C(object):
       alpha = 1.0
       beta = [3, 4]
    c = C()
    df = shelve.open('test.shelve', 'c')
    df['a'] = a
    df['b'] = b
    df['c'] = c
    df.sync()
    exit()
    

Following the above example, recovering data:

    
    
    import shelve
    import numpy as np
    class C():
       alpha = 1.0
       beta = [3, 4]
    df = shelve.open('test.shelve')
    print(df['a'])
    print(df['b'])
    print(df['c'].alpha)
    

