PyTorch LSTM States
Consider the following code snipped:

lstm = nn.LSTM(10, 5, batch_first=True)
states = (torch.rand(1, 1, 5), torch.rand(1, 1, 5))
h, states = lstm(torch.rand(1, 1, 10), states)
print('h:')
print(h)
print('states[0]:')
print(states[0])


Output:

h:
tensor([[[0.2808, 0.3357, 0.1290, 0.1413, 0.2648]]], grad_fn=&lt;TransposeBackward0&gt;)
states[0]:
tensor([[[0.2808, 0.3357, 0.1290, 0.1413, 0.2648]]], grad_fn=&lt;ViewBackward&gt;)




Because I have to hand over states as parameter for the forward() anyway I'd prefer using states[0] over h.

I've just noticed that the grad_fn is different, therefore I'm wondering if it makes any difference for backpropagation if using h or states for further computation of the outputs.

I can hardly imagine that there is a difference, so I'd probably just continue with states[0] but I also would like to understand why it is different.

Thanks in advance!

It's best practice and more intuitive to use h or (often called output) since states are meant to be passed into the lstm for internal use (think of tensorflow's dynamic_rnn to see why this would be the case.

That said you are correct that it actually doesn't make a difference. I'm not sure why the grad_fns are different, but empirically they function the same:

import torch
from torch import nn

lstm = nn.LSTM(10, 5, batch_first=True)
state = (torch.rand(1, 1, 5), torch.rand(1, 1, 5))
inp = torch.rand(1, 1, 10)
h, states = lstm(inp, state)

param = next(lstm.parameters())

l1 = h.sum()
l1.backward(retain_graph=True)
g1 = param.grad.clone()

param.grad.zero_()

l2 = states[0].sum()
l2.backward(retain_graph=True)
g2 = param.grad.clone()

print((g1 == g2).all())  # 1

