title: When should one set the staircase is True when decaying the learning rate in TensorFlow?
body: <p>Recall that when <a href="https://www.tensorflow.org/versions/r0.9/api_docs/python/train.html#decaying-the-learning-rate" rel="nofollow">exponentially decaying</a> the learning rate in TensorFlow one does:</p>
  
  <blockquote>
    <p>decayed_learning_rate = learning_rate *
                            decay_rate ^ (global_step / decay_steps)</p>
  </blockquote>
  
  <p>the docs mention this staircase option as:</p>
  
  <blockquote>
    <p>If the argument staircase is True, then global_step /decay_steps is an
    integer division and the decayed learning rate follows a staircase
    function.</p>
  </blockquote>
  
  <p>when is it better to decay every X number of steps and follow at stair case function rather than a smoother version that decays more and more with every step?</p>