title: Backpropagation Through Time (BPTT) of LSTM
body: <p>I am currently trying to understand the BPTT for LSTM in TensorFlow. I get that the parameter "num_steps" is used for the range that the RNN is rolled out and the Error backpropagated. I got a general question of how this works.</p>
  
  <p>For reference a repitition of the formulas. I'm referring to:
  <a href="https://i.stack.imgur.com/3zKXt.png" rel="nofollow noreferrer">Formulas LSTM</a> (<a href="https://arxiv.org/abs/1506.00019" rel="nofollow noreferrer">https://arxiv.org/abs/1506.00019</a>)</p>
  
  <p><strong>Question:</strong>
  What paths are backpropagated that many steps? The constant error carousel is created by the formula 5, and the derivate for backpropagation (s(t)->s(t-1)) is 1 for all timesteps. This is why LSTMs capture long range dependencies. I get confused with the dependencies of g(t), i(t), f(t) and o(t) of h(t-1). In words: The current gates do not just depend on the input, but also on the last hidden state.</p>
  
  <p><strong>Doesn't this dependency lead to the exploding/vanishing gradients problem again?</strong></p>
  
  <p>If I backpropagate along these connections I get gradients that are not one. Peephole connections essentially lead to the same problem.</p>
  
  <p>Thanks for your help!</p>