title: How to apply gradient clipping in TensorFlow?
body: <p>Considering the <a href="https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3%20-%20Neural%20Networks/recurrent_network.py">example code</a>.</p>
  
  <p>I would like to know How to apply gradient clipping on this network on the RNN where there is a possibility of exploding gradients.</p>
  
  <pre><code>tf.clip_by_value(t, clip_value_min, clip_value_max, name=None)
  </code></pre>
  
  <p>This is an example that could be used but where do I introduce this ?
  In the def of RNN </p>
  
  <pre><code>    lstm_cell = rnn_cell.BasicLSTMCell(n_hidden, forget_bias=1.0)
      # Split data because rnn cell needs a list of inputs for the RNN inner loop
      _X = tf.split(0, n_steps, _X) # n_steps
  tf.clip_by_value(_X, -1, 1, name=None)
  </code></pre>
  
  <p>But this doesn't make sense as the tensor _X is the input and not the grad what is to be clipped? </p>
  
  <p>Do I have to define my own Optimizer for this or is there a simpler option?</p>