title: learning rate decay in LSTM
body: <p>I am currently reproducing the code for char-RNN described in <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/" rel="nofollow">http://karpathy.github.io/2015/05/21/rnn-effectiveness/</a>. There are codes already implemented in tensorflow and the code I am referring to is at <a href="https://github.com/sherjilozair/char-rnn-tensorflow/blob/master/train.py" rel="nofollow">https://github.com/sherjilozair/char-rnn-tensorflow/blob/master/train.py</a> I am having a question for the learning rate decay.In the code the optimizer is defined as an AdamOptimizer. When I went through the code, I saw a line as following:</p>
  
  <pre><code>for e in range(args.num_epochs):
          sess.run(tf.assign(model.lr, args.learning_rate * (args.decay_rate ** e)))
  </code></pre>
  
  <p>which adjusts the learning rate by a decay constant. 
  My question is: isn't Adam optimizer making us able to control the learning rate? Why do we still use a decay rate with respect to learning rate here? </p>