title: Truncated backpropagation in PyTorch (code check)
body: <p>I am trying to implement truncated backpropagation through time in PyTorch, for the simple case where <code>K1=K2</code>. I have an implementation below that produces reasonable output, but I just want to make sure it is correct. When I look online for PyTorch examples of TBTT, they do inconsistent things around detaching the hidden state and zeroing out the gradient, and the ordering of these operations. Please let me know if I have made a mistake.</p>
  <p>In the code below, <code>H</code> maintains the current hidden state, and <code>model(weights, H, x)</code> outputs the prediction and the new hidden state.</p>
  <pre><code>while i &lt; NUM_STEPS:
      # Grab x, y for ith datapoint
      x = data[i]
      target = true_output[i]
  
      # Run model
      output, new_hidden = model(weights, H, x)
      H = new_hidden
  
      # Update running error
      error += (output - target)**2
  
      if (i+1) % K == 0:
          # Backpropagate
          error.backward()
          opt.step()
          opt.zero_grad()
          error = 0
          H = H.detach()
  
      i += 1
  </code></pre>