Best data type (in terms of speed/RAM) for millions of pairs of a single int paired with a batch (2 to 100) of ints
Use numpy. It us the most efficient and you can use it easily with a machine learning model.
I have about 15 million pairs that consist of a single int, paired with a batch of (2 to 100) other ints. 
If it makes a difference, the ints themselve range from 0 to 15 million. 
I have considered using: 
Pandas, storing the batches as python lists
Numpy, where the batch is stored as it's own numpy array (since numpy doesn't allow variable length rows in it's 2D data structures)
Python List of Lists. 
I also looked at Tensorflow tfrecords but not too sure about this one. 
I only have about 12 gbs of RAM. I will also be using to train over a machine learning algorithm so