title: TensorFlow: Neural Network accuracy always 100% on train and test sets
body: <p>I created a TensorFlow neural network that has 2 hidden layers with 10 units each using ReLU activations and Xavier Initialization for the weights. The output layer has 1 unit outputting binary classification (0 or 1) using the sigmoid activation function to classify whether it believes a passenger on the titanic survived based on the input features. </p>
  
  <p>(The only code omitted is the load_data function which populates the variables X_train, Y_train, X_test, Y_test used later in the program)</p>
  
  <p>Parameters</p>
  
  <pre><code># Hyperparams 
  learning_rate = 0.001
  lay_dims = [10,10, 1]
  
  # Other params
  m = X_train.shape[1] 
  n_x = X_train.shape[0]
  n_y = Y_train.shape[0]
  </code></pre>
  
  <p>Inputs </p>
  
  <pre><code>X = tf.placeholder(tf.float32, shape=[X_train.shape[0], None], name="X")
  norm = tf.nn.l2_normalize(X, 0) # normalize inputs
  
  Y = tf.placeholder(tf.float32, shape=[Y_train.shape[0], None], name="Y")
  </code></pre>
  
  <p>Initialize Weights &amp; Biases</p>
  
  <pre><code>W1 = tf.get_variable("W1", [lay_dims[0],n_x], initializer=tf.contrib.layers.xavier_initializer())
  b1 = tf.get_variable("b1", [lay_dims[0],1], initializer=tf.zeros_initializer())
  
  W2 = tf.get_variable("W2", [lay_dims[1],lay_dims[0]], initializer=tf.contrib.layers.xavier_initializer())
  b2 = tf.get_variable("b2", [lay_dims[1],1], initializer=tf.zeros_initializer())
  
  W3 = tf.get_variable("W3", [lay_dims[2],lay_dims[1]], initializer=tf.contrib.layers.xavier_initializer())
  b3 = tf.get_variable("b3", [lay_dims[2],1], initializer=tf.zeros_initializer())
  </code></pre>
  
  <p>Forward Prop</p>
  
  <pre><code>Z1 = tf.add(tf.matmul(W1,X), b1)
  A1 = tf.nn.relu(Z1)
  
  Z2 = tf.add(tf.matmul(W2,A1), b2)
  A2 = tf.nn.relu(Z2)
  
  Y_hat = tf.add(tf.matmul(W3,A2), b3)
  </code></pre>
  
  <p>BackProp</p>
  
  <pre><code>cost = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=tf.transpose(Y_hat), labels=tf.transpose(Y)))
  optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
  </code></pre>
  
  <p>Session</p>
  
  <pre><code># Initialize
  init = tf.global_variables_initializer()
  
  with tf.Session() as sess:
      # Initialize
      sess.run(init)
  
      # Normalize Inputs
      sess.run(norm, feed_dict={X:X_train, Y:Y_train})
  
      # Forward/Backprob and update weights
      for i in range(10000):
          c, _ = sess.run([cost, optimizer], feed_dict={X:X_train, Y:Y_train})
          if i % 100 == 0:
              print(c)
  
      correct_prediction = tf.equal(tf.argmax(Y_hat), tf.argmax(Y))
  
      accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
  
      print("Training Set:", sess.run(accuracy, feed_dict={X: X_train, Y: Y_train}))
      print("Testing Set:", sess.run(accuracy, feed_dict={X: X_test, Y: Y_test}))
  </code></pre>
  
  <p>After running running 10,000 epochs of training, the cost goes down each time so it shows that the learning_rate is okay and that the cost function appears normal. However, after training, all of my Y_hat values (predictions on the training set) are 1 (predicting the passenger survived). So basically the prediction just outputs y=1 for every training example. </p>
  
  <p>Also, when I run tf.argmax on Y_hat, the result is a matrix of all 0's. The same thing is happening when tf.argmax is applied to Y (ground truth labels) which is odd because Y consists of all the correct labels for the training examples.</p>
  
  <p>Any help is greatly appreciated. Thanks.</p>