title: Theano - logistic regression example weight vector becomes NaN?
body: <p>I am doing a tutorial (code <a href="https://github.com/Newmu/Theano-Tutorials/blob/master/2_logistic_regression.py" rel="nofollow">here</a>) and video <a href="https://www.youtube.com/watch?v=S75EdAcXHKk" rel="nofollow">here</a> (13:00 minutes in).  </p>
  
  <p>My only change is using the mnist training set from a different location (creating a one-hot encoding) but it is not working.  I literally copy-pasted all the code (except for the mnist loading) in this example.  Here is the code:</p>
  
  <pre><code>import theano
  from theano import tensor as T
  import numpy as np  
  from sklearn.datasets import fetch_mldata
  mnist = fetch_mldata("MNIST Original")
  trX, teX, trY_digit, teY_digit = train_test_split(mnist.data, mnist.target, test_size=.4)
  
  #Get one-hot encoding
  enc = OneHotEncoder()
  enc.fit([[n] for n in range(10)])
  trY, teY = sparse_to_floatX(enc.transform(trY_digit[:,newaxis])), sparse_to_floatX(enc.transform(teY_digit[:,newaxis]))
  
  def floatX(X):
      return np.asarray(X, dtype=theano.config.floatX)
  
  def init_weights(shape):
      return theano.shared(floatX(np.random.randn(*shape) * 0.1))
  
  def model(X, w):
      return T.nnet.softmax(T.dot(X, w))
  
  X = T.fmatrix()
  Y = T.fmatrix()
  
  w = init_weights((784, 10))
  
  py_x = model(X, w)
  y_pred = T.argmax(py_x, axis=1)
  
  cost = T.mean(T.nnet.categorical_crossentropy(py_x, Y))
  gradient = T.grad(cost=cost, wrt=w)
  update = [[w, w - gradient * 0.05]]
  
  train = theano.function(inputs=[X, Y], outputs=cost, updates=update, allow_input_downcast=True)
  predict = theano.function(inputs=[X], outputs=y_pred, allow_input_downcast=True)
  
  for i in range(10):
      print w.get_value()
      cost = train(trX, trY)
      print i, predict(teX)
  </code></pre>
  
  <p>The weight vector updates once, and becomes all <code>NaN</code> on the second update.  I am very new to theano, but I am looking for tips to figure this out, especially if someone has already done this tutorial.</p>
  
  <p><strong>UPDATE</strong>.
  It looks like the gradient is the issue.</p>
  
  <p>When I add this</p>
  
  <pre><code>the_grad = T.sum(gradient)
  f_grad = theano.function(inputs=[X, Y], outputs=the_grad, allow_input_downcast=True)
  print f_grad(trX, trY)
  </code></pre>
  
  <p>It prints <code>NaN</code>.  This appears to be the correct usage of <code>T.grad</code> though.</p>
  
  <p><strong>UPDATE 2.</strong>
  When I change the cost function to this:</p>
  
  <pre><code>cost = T.mean(T.sum(T.sqr(py_x - Y), axis=1), axis=0)
  </code></pre>
  
  <p>It is working now but I only have 70% accuracy which is really bad.</p>
  
  <p><strong>UPDATE 3.</strong>
  I downloaded the MNIST data used in the tutorial and it worked with 92% accuary.</p>
  
  <p>I am not sure why my first mnist datasource was dying with the crossentropy cost, and then performing really poor with mean squared error cost function.</p>