Storing a large table on disk, with fast retrieval of a specified subset to np.ndarray
You could create a region reference dataset where each element relates to one of the ~2000 identifiers.
Then the Python code to reference a particular identifier would look like this: 
  reg_ref - reg_ref_dset[identifier]
  mysub = data_dset[reg_ref]
I need to store a table on disk, and be able to retrieve a subset of that table into a numpy.ndarray very fast. What's the best way to do that? I don't mind spending the time to preprocess this dataset before storing it on disk, since it won't be changed once it's created.
I'd prefer not to write any C code, and instead rely on existing python libraries. I am considering HDF5 (with either pytables or h5py), sqlite, numpy's memmap, or a custom binary file format.
For a custom file, I would sort the rows by the identifier, and add to the file a table of contents which, for every identifier, would specify the beginning and ending file offsets that encompass the data related to this identifier. This would probably be very fast in terms of I/O, but at a cost of using python rather than C code (since I don't think there's a library that does precisely that).
Details:
~100 million rows, ~5 columns of float and str data. One of the columns contains 100,000 different identifiers (so there are about 1000 rows per identifier). The subset to be retrieved is always specified by a set of identifiers (usually I need to retrieve ~2000 identifiers, so ~2% of the entire dataset).
Python 3.4, Linux, SSD drive (so random access is as fast as sequential).