Index pytorch 4d tensor by values in 2d tensor
I have two pytorch tensors:


X with shape (A, B, C, D)
I with shape (A, B)


Values in I are integers in range [0, C).



What is the most efficient way to get tensor Y with shape (A, B, D), such that:

Y[i][j][k] = X[i][j][ I[i][j] ][k]


You probably want to use torch.gather for the indexing and expand to adjust I to the required size:

eI = I[..., None, None].expand(-1, -1, 1, X.size(3))  # make eI the same for the last dimension
Y = torch.gather(X, dim=2, index=eI).squeeze()


testing the code:

A = 3 
B = 4 
C = 5 
D = 7

X = torch.rand(A, B, C, D)
I = torch.randint(0, C, (A, B), dtype=torch.long)

eI = I[..., None, None].expand(-1, -1, 1, X.size(3))
Y = torch.gather(X, dim=2, index=eI).squeeze()

# manually gather
refY = torch.empty(A, B, D)
for i in range(A):
    for j in range(B):
        refY[i, j, :] = X[i, j, I[i,j], :]

(refY == Y).all()
# Out[]: tensor(1, dtype=torch.uint8)

