title: Logistic Regression and Scipy Optimization with fmin_bfgs
body: <p>I am trying to implement Logistic Regression and I am using Scipy's Optimize module to find the optimized theta values. I am able to get the correct value when using the fmin function. But I would like to do so when using the fmin_bfgs function which requires the gradient. </p>
  
  <p>Here is a code snippet:</p>
  
  <pre><code>#Returns Cost of current theta values and gradient of cost w.r.t theta.
  def costFunction(theta, X, y, _lambda):
  
      #Initializes useful variables
      m = len(y)
      grad = np.zeros((np.shape(theta)))
  
      #Solves for Hypothesis for input X.
      h_x = Sigmoid(np.dot(X,theta))
  
      #Reshaped because numpy kept returning row vector, not column.
      h_x = h_x.reshape((m,1))
  
      #Cost is broken up into terms for simplicity. 
      term1 = -y*np.log(h_x)
      term2 = (1-y)*np.log((1-h_x))
  
  
  
      #Regularized Cost FUnction
      J = (1/m) * sum(term1-term2) + (_lambda/(2*m)) * sum(np.square(theta[1:][:]))
  
      #Gradient for Theta1
      grad_reg = (_lambda/m)*theta[1:]
  
      #Combines gradient for Theta1 and onward. 
      grad = (1/m)* (np.dot(np.transpose(X),(h_x-y))) + np.vstack(([0],grad_reg)) 
  
      return J,grad 
  
  
  
  
  
  #Finds Optimal Value for theta
  cost, grad = costFunction(initial_theta, X,y, _lambda)
  opt_theta = fmin_bfgs(cost,x0=initial_theta,fprime=grad, args = (X,y,_lambda))
  </code></pre>
  
  <p>The error I get is  <code>'numpy.ndarray' object is not callable</code>which comes from the <code>function_wrapper</code> function in the optimize module. I even tried returning the gradient and the cost in two different functions, but got a <code>vstack</code> error of some sort(if that matters/helps at all). </p>
  
  <p>As far as I can see I have provided what the optimization function ask for. </p>
  
  <p><strong>EDIT/UPDATE:</strong> I realized that the error I was getting was because I was passing the cost, and grad numpy arrays as parameters when it was expecting a function to return those parameters. I realize that I could create a wrapper function? to get both of those values without using two separate functions, but for temp purposes, I changed costFunction so it only returns the cost, and created a whole new function,<code>Grad()</code>, (with identical code though) that only returns the grad. This gives me the  <code>all the input array dimensions except for the concatenation axis must match exactly</code> vstack error again. </p>