title: fmin_cg: Desired error not necessarily achieved due to precision loss
body: <p>I have the following code to minimize the Cost Function with its gradient. </p>
  
  <pre><code>def trainLinearReg( X, y, lamda ):
      # theta = zeros( shape(X)[1], 1 )
      theta = random.rand( shape(X)[1], 1 ) # random initialization of theta
  
      result = scipy.optimize.fmin_cg( computeCost, fprime = computeGradient, x0 = theta, 
                                       args = (X, y, lamda), maxiter = 200, disp = True, full_output = True )
      return result[1], result[0]
  </code></pre>
  
  <p>But I am having this warning:</p>
  
  <pre><code>Warning: Desired error not necessarily achieved due to precision loss.
           Current function value: 8403387632289934651424768.000000
           Iterations: 0
           Function evaluations: 15
           Gradient evaluations: 3
  </code></pre>
  
  <p>My <code>computeCost</code> and <code>computeGradient</code> are defined as </p>
  
  <pre><code>def computeCost( theta, X, y, lamda ):
      theta = theta.reshape( shape(X)[1], 1 )
      m     = shape(y)[0]
      J     = 0
      grad  = zeros( shape(theta) )
  
      h = X.dot(theta)
      squaredErrors = (h - y).T.dot(h - y)
      # theta[0] = 0.0
      J = (1.0 / (2 * m)) * (squaredErrors) + (lamda / (2 * m)) * (theta.T.dot(theta))
  
      return J[0]
  
  def computeGradient( theta, X, y, lamda ):
      theta = theta.reshape( shape(X)[1], 1 )
      m     = shape(y)[0]
      J     = 0
      grad  = zeros( shape(theta) )
  
      h = X.dot(theta)
      squaredErrors = (h - y).T.dot(h - y)
      # theta[0] = 0.0
      J = (1.0 / (2 * m)) * (squaredErrors) + (lamda / (2 * m)) * (theta.T.dot(theta))
      grad = (1.0 / m) * (X.T.dot(h - y)) + (lamda / m) * theta
  
      return grad.flatten()
  </code></pre>
  
  <p>I have reviewed these similar questions:</p>
  
  <p><a href="https://scicomp.stackexchange.com/questions/2004/scipy-optimize-fmin-bfgs-desired-error-not-necessarily-achieved-due-to-precisi">scipy.optimize.fmin_bfgs: “Desired error not necessarily achieved due to precision loss”</a></p>
  
  <p><a href="https://stackoverflow.com/questions/31474895/scipy-optimize-fmin-cg-desired-error-not-necessarily-achieved-due-to-precisio">scipy.optimize.fmin_cg: &quot;&#39;Desired error not necessarily achieved due to precision loss.&#39;</a></p>
  
  <p><a href="https://stackoverflow.com/questions/24767191/scipy-is-not-optimizing-and-returns-desired-error-not-necessarily-achieved-due">scipy is not optimizing and returns &quot;Desired error not necessarily achieved due to precision loss&quot;</a></p>
  
  <p>But still cannot have the solution to my problem. How to let the minimization function process converge instead of being stuck at first? </p>
  
  <hr>
  
  <p><strong>ANSWER:</strong></p>
  
  <p>I solve this problem based on @lejlot 's comments below. 
  He is right. The data set <code>X</code> is to large since I did not properly return the correct normalized value to the correct variable. Even though this is a small mistake, it indeed can give you the thought where should we look at when encountering such problems.  The Cost Function value is too large leads to the possibility that there are some wrong with my data set. </p>
  
  <p>The previous wrong one:</p>
  
  <pre><code>X_poly            = polyFeatures(X, p)
  X_norm, mu, sigma = featureNormalize(X_poly)
  X_poly            = c_[ones((m, 1)), X_poly]
  </code></pre>
  
  <p>The correct one: </p>
  
  <pre><code>X_poly            = polyFeatures(X, p)
  X_poly, mu, sigma = featureNormalize(X_poly)
  X_poly            = c_[ones((m, 1)), X_poly]
  </code></pre>
  
  <p>where <code>X_poly</code> is actually used in the following traing as </p>
  
  <pre><code>cost, theta = trainLinearReg(X_poly, y, lamda)
  </code></pre>