title: Gradient descent convergence How to decide convergence?
body: <p>I learnt gradient descent through online resources (namely machine learning at coursera). However the information provided only said to repeat gradient descent until it converges.</p>
  
  <p>Their definition of convergence was to use a graph of the cost function relative to the number of iterations and watch when the graph flattens out. Therefore I assume that I would do the following:</p>
  
  <pre><code>if (change_in_costfunction &gt; precisionvalue) {
            repeat gradient_descent
  } 
  </code></pre>
  
  <p>Alternatively, I was wondering if another way to determine convergence is to watch the coefficient approach it's true value:</p>
  
  <pre><code>if (change_in_coefficient_j &gt; precisionvalue) {
            repeat gradient_descent_for_j
  } 
  ...repeat for all coefficients
  </code></pre>
  
  <p>So is convergence based on the cost function or the coefficients? And how do we determine the precision value? Should it be a % of the coefficient or total cost function?</p>