title: Neural network always predicts the same class
body: <p>I'm trying to implement a neural network that classifies images into one of the two discrete categories. The problem is, however, that it currently always predicts 0 for any input and I'm not really sure why.</p>
  
  <p>Here's my feature extraction method:</p>
  
  <pre><code>def extract(file):
      # Resize and subtract mean pixel
      img = cv2.resize(cv2.imread(file), (224, 224)).astype(np.float32)
      img[:, :, 0] -= 103.939
      img[:, :, 1] -= 116.779
      img[:, :, 2] -= 123.68
      # Normalize features
      img = (img.flatten() - np.mean(img)) / np.std(img)
  
      return np.array([img])
  </code></pre>
  
  <p>Here's my gradient descent routine:</p>
  
  <pre><code>def fit(x, y, t1, t2):
      """Training routine"""
      ils = x.shape[1] if len(x.shape) &gt; 1 else 1
      labels = len(set(y))
  
      if t1 is None or t2 is None:
          t1 = randweights(ils, 10)
          t2 = randweights(10, labels)
  
      params = np.concatenate([t1.reshape(-1), t2.reshape(-1)])
      res = grad(params, ils, 10, labels, x, y)
      params -= 0.1 * res
  
      return unpack(params, ils, 10, labels)
  </code></pre>
  
  <p>Here are my forward and back(gradient) propagations:</p>
  
  <pre><code>def forward(x, theta1, theta2):
      """Forward propagation"""
  
      m = x.shape[0]
  
      # Forward prop
      a1 = np.vstack((np.ones([1, m]), x.T))
      z2 = np.dot(theta1, a1)
  
      a2 = np.vstack((np.ones([1, m]), sigmoid(z2)))
      a3 = sigmoid(np.dot(theta2, a2))
  
      return (a1, a2, a3, z2, m)
  
  def grad(params, ils, hls, labels, x, Y, lmbda=0.01):
      """Compute gradient for hypothesis Theta"""
  
      theta1, theta2 = unpack(params, ils, hls, labels)
  
      a1, a2, a3, z2, m = forward(x, theta1, theta2)
      d3 = a3 - Y.T
      print('Current error: {}'.format(np.mean(np.abs(d3))))
  
      d2 = np.dot(theta2.T, d3) * (np.vstack([np.ones([1, m]), sigmoid_prime(z2)]))
      d3 = d3.T
      d2 = d2[1:, :].T
  
      t1_grad = np.dot(d2.T, a1.T)
      t2_grad = np.dot(d3.T, a2.T)
  
      theta1[0] = np.zeros([1, theta1.shape[1]])
      theta2[0] = np.zeros([1, theta2.shape[1]])
  
      t1_grad = t1_grad + (lmbda / m) * theta1
      t2_grad = t2_grad + (lmbda / m) * theta2
  
      return np.concatenate([t1_grad.reshape(-1), t2_grad.reshape(-1)])
  </code></pre>
  
  <p>And here's my prediction function:</p>
  
  <pre><code>def predict(theta1, theta2, x):
      """Predict output using learned weights"""
      m = x.shape[0]
  
      h1 = sigmoid(np.hstack((np.ones([m, 1]), x)).dot(theta1.T))
      h2 = sigmoid(np.hstack((np.ones([m, 1]), h1)).dot(theta2.T))
  
      return h2.argmax(axis=1)
  </code></pre>
  
  <p>I can see that the error rate is gradually decreasing with each iteration, generally converging somewhere around 1.26e-05.</p>
  
  <p>What I've tried so far:</p>
  
  <ol>
  <li>PCA</li>
  <li>Different datasets (Iris from sklearn and handwritten numbers from Coursera ML course, achieving about 95% accuracy on both). However, both of those were processed in a batch, so I can assume that my general implementation is correct, but there is something wrong with either how I extract features, or how I train the classifier.</li>
  <li>Tried sklearn's SGDClassifier and it didn't perform much better, giving me a ~50% accuracy. So something wrong with the features, then?</li>
  </ol>
  
  <p><strong>Edit</strong>:
  An average output of h2 looks like the following:</p>
  
  <pre><code>[0.5004899   0.45264441]
  [0.50048522  0.47439413]
  [0.50049019  0.46557124]
  [0.50049261  0.45297816]
  </code></pre>
  
  <p>So, very similar sigmoid outputs for all validation examples.</p>