title: LSTM in Pytorch
body: <p>I'm new to PyTorch. I came across some this <a href="https://github.com/yunjey/pytorch-tutorial/blob/master/tutorials/02-intermediate/recurrent_neural_network/main.py#L38-L56" rel="nofollow noreferrer">GitHub repository (link to full code example)</a> containing various different examples.</p>
  
  <p>There is also an example about LSTMs, this is the Network class:</p>
  
  
  
  <pre class="lang-python prettyprint-override"><code># RNN Model (Many-to-One)
  class RNN(nn.Module):
      def __init__(self, input_size, hidden_size, num_layers, num_classes):
          super(RNN, self).__init__()
          self.hidden_size = hidden_size
          self.num_layers = num_layers
          self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)
          self.fc = nn.Linear(hidden_size, num_classes)
  
      def forward(self, x):
          # Set initial states 
          h0 = Variable(torch.zeros(self.num_layers, x.size(0), self.hidden_size)) 
          c0 = Variable(torch.zeros(self.num_layers, x.size(0), self.hidden_size))
  
          # Forward propagate RNN
          out, _ = self.lstm(x, (h0, c0))  
  
          # Decode hidden state of last time step
          out = self.fc(out[:, -1, :])  
          return out
  </code></pre>
  
  <p>So my question is about the following lines:</p>
  
  <pre class="lang-python prettyprint-override"><code>h0 = Variable(torch.zeros(self.num_layers, x.size(0), self.hidden_size)) 
  c0 = Variable(torch.zeros(self.num_layers, x.size(0), self.hidden_size))
  </code></pre>
  
  <p>As far as I understand it, <code>forward()</code> is called for every training example. But this would mean, that the hidden state and cell state would be resettet i.e. replaced with a matrix of zeros on every training example.</p>
  
  <p>The names <code>h0</code> and <code>c0</code> indicate that this is only the hidden/cell state at t=0, but why then are theses zeros matrices handed over to the lstm with every training example?</p>
  
  <p>Even if they are just ignored after the first call, it would not be a very nice solution.</p>
  
  <p>When testing the code it states an accuracy of 97% on the MNIST set, so it seems to work this way, but it doesn't make sense to me.</p>
  
  <p>Hope someone can help me out with this.</p>
  
  <p>Thanks in advance!</p>