What's an appropriate datastore for variable length sequence data for PyTorch consumption?
What I've opted for at the moment is packing all of the samples into a single HDF5 table buffer, and keeping a separate table with metadata that tracks each individual sequence's buffer position and length. This works, but I won't be marking this answer as correct because I'm not satisfied with it. This storage method is very poorly suited to editing, and it's vulnerable to loss if a bug were to cause the tables to become out of sync.