title: Neural Network With One Hidden Layer Cannot Be Trained
body: <p>I was trying to implement an NN with one hidden layer by using TensorFlow to recognize MNIST handwritten digits. I was using gradient descent method to train the NN. However, it seems that my training toward the NN did not work at all, as the testing accuracy did not change at all during the training process.</p>
  
  <p>Can anyone help me figure out what went wrong?</p>
  
  <p>Here is my code.</p>
  
  <pre><code>import tensorflow as tf
  from tensorflow.examples.tutorials.mnist import input_data
  
  mnist = input_data.read_data_sets("MNIST_data", one_hot=True)
  
  batch_size = 100
  
  n_batch = mnist.train.num_examples // batch_size
  
  x = tf.placeholder(tf.float32, [None, 784])
  y = tf.placeholder(tf.float32, [None, 10])
  
  #First layer of the NN
  W1 = tf.Variable(tf.zeros([784,10]))
  b1 = tf.Variable(tf.zeros([10]))
  out1 = tf.nn.softmax(tf.matmul(x, W1) + b1)
  
  #Second layer of the NN
  W2 = tf.Variable(tf.zeros([10,10]))
  b2 = tf.Variable(tf.zeros([10]))
  prediction = tf.nn.softmax(tf.matmul(out1, W2) + b2)
  
  loss = tf.reduce_mean(tf.square(y - prediction))
  
  train_step = tf.train.GradientDescentOptimizer(0.1).minimize(loss)
  
  init = tf.global_variables_initializer()
  
  correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(prediction, 1))
  
  accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
  
  with tf.Session() as sess:
      sess.run(init)
      for epoch in range(101):
          for batch in range(n_batch):
              batch_xs, batch_ys = mnist.train.next_batch(batch_size)
              sess.run(train_step, feed_dict={x:batch_xs, y:batch_ys})
  
          acc = sess.run(accuracy, feed_dict={x:mnist.test.images, y:mnist.test.labels})
          print("Iter " + str(epoch) + ", Testing Accuracy " + str(acc))
  </code></pre>