title: Tensorflow: How to determine the value of clip_norm when using clip_by_norm or clip_by_global_norm?
body: <p>I am trying to find out how to determine the value of <code>clip_norm</code> when using <a href="https://www.tensorflow.org/api_docs/python/tf/clip_by_norm" rel="nofollow noreferrer">clip_by_norm</a> or <a href="https://www.tensorflow.org/api_docs/python/tf/clip_by_global_norm" rel="nofollow noreferrer">clip_by_global_norm</a> with Tensorboard.</p>
  
  <p>In Tensorflow, we can use the optimizer to <code>compute_gradients</code> to obtain the gradient and add it to the <a href="https://www.tensorflow.org/api_docs/python/tf/summary/histogram" rel="nofollow noreferrer">tf.summary.histogram</a>. In Tensorboard, we can observe the range of the gradient in the <code>DISTRIBUTIONS</code> tab. Then we can determine the <code>clip_value</code> range when using <a href="https://www.tensorflow.org/api_docs/python/tf/clip_by_value" rel="nofollow noreferrer">clip_by_value</a>.</p>
  
  <p>On the other hand, how shall I calculate/display the norm or global norm in Tensorboard  so that I can determine the value of <code>clip_norm</code>?</p>