title: State Normalization of RNNs
body: <p>Perhaps a question better posed to Computer Science or Cross Validated?</p>
  
  <hr>
  
  <p>I'm beginning some work with LSTM on sequences of arbitrary length and one problem I'm experiencing and that I haven't seen addressed, is that my network seems to have developed a couple parameters that grow linearly (perhaps as a measure of time?).</p>
  
  <p>The obvious issue with this is that the training data is bounded at a sequence of length <code>x</code> and so the network grows this parameter reasonably up until tilmestep <code>x</code>. But after that, the network will eventually NAN because values are getting too extreme.</p>
  
  <p>Has anyone read anything about the normalization of stabilization of states over time?</p>
  
  <p>Any suggestions would be much appreciated.</p>