Store a multi-dim tensor on disk and read from offset
I have a multi-dimensional tensor like this
tensor([[ 0.5599,  0.4593,  0.0580,  ..., -0.2404,  0.1144, -0.5047],
        [ 0.1545,  0.3332,  0.3836,  ...,  0.2483, -0.0849, -0.2216],
        [ 0.4513,  0.0115,  0.0801,  ..., -0.8038,  0.2350, -0.3261],
        ...,
        [-0.4387,  0.3028, -0.0510,  ..., -0.4966, -0.1606,  0.2933],
        [ 0.0312,  0.2351, -0.0397,  ..., -0.5401, -0.0554, -0.1552],
        [-0.3732, -0.0460,  0.0698,  ..., -0.2963, -0.3514, -0.3815]],
       device='cuda:0', requires_grad=True)
Size of entity embeddings:  torch.Size([14951, 400])

What is the best way to store this tensor on disk, and only index a row of this multi-dimensional tensor (i.e. say the last row, [-0.3732, -0.0460,  0.0698,  ..., -0.2963, -0.3514, -0.3815] and bring it to memory in python?

The answer depends on what you are trying to maximize/minimize.
You could define &quot;Best way to store&quot; as the fastest write, the fastest read, the smallest file, ...
But given your constraints I think HDFS should be a good candidate.
Pandas allows you to save as HDFS format with the df.to_hdf() function.
You can also look at H5py to deal with HDFS files.
