title: Python - Tensorflow, binary classification, always predicting 0
body: <p>I am just starting out with Tensorflow, trying to create a classic neural net for binary classification.</p>
  
  <pre><code># Loading Dependencies
  
  import math
  import numpy as np
  import pandas as pd
  import tensorflow as tf
  from tensorflow.python.framework import ops
  from sklearn.model_selection import train_test_split
  import matplotlib.pyplot as plt
  
  seed = 1234
  tf.set_random_seed(seed)
  np.random.seed(seed)
  
  # Load and Split data
  data = pd.read_json(file)
  X = data["X"]
  y = data["y"]
  X = X.astype(np.float32)
  y = y.astype(np.float32)
  
  X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size = 0.3)
  
  X_train = X_train.reshape(X_train.shape[0], -1).T
  y_train = y_train.values.reshape((1, y_train.shape[0]))
  X_valid = X_valid.reshape(X_valid.shape[0], -1).T
  y_valid = y_valid.values.reshape((1, y_valid.shape[0]))
  
  print("X Train: ", X_train.shape)
  print("y Train: ", y_train.shape)
  print("X Dev: ", X_valid.shape)
  print("y Dev: ", y_valid.shape)
  </code></pre>
  
  <p>X Train:  (16875, 1122) <br />
  y Train:  (1, 1122) <br />
  X Dev:  (16875, 482) <br />
  y Dev:  (1, 482) <br /></p>
  
  <p>The training data contains float numbers, while the labels are just 0 or 1. However, these are also converted to float because I had some issues in the past.</p>
  
  <p>Initializing the parameters</p>
  
  <pre><code>def initialize_parameters(layer_dimensions):
      tf.set_random_seed(seed)
      layers_count = len(layer_dimensions)
      parameters = {}
  
      for layer in range(1, layers_count):
          parameters['W' + str(layer)] = tf.get_variable('W' + str(layer), 
                                                     [layer_dimensions[layer], layer_dimensions[layer - 1]], 
                                                     initializer = tf.contrib.layers.xavier_initializer(seed = seed))
  
          parameters['b' + str(layer)] = tf.get_variable('b' + str(layer), 
                                                     [layer_dimensions[layer], 1], 
                                                     initializer = tf.zeros_initializer())
  
      return parameters
  </code></pre>
  
  <p>Shapes are: <br />
  W1 - (50, 16875) <br />
  W2 - (25, 50) <br />
  W3 - (10, 25) <br />
  W4 - (5, 10) <br />
  W5 - (1, 5) <br />
  b1 - (50, 1) <br />
  b2 - (25, 1) <br />
  b3 - (10, 1) <br />
  b4 - (5, 1) <br />
  b5 - (1, 1) <br /></p>
  
  <p>I am specifying the number and the dimension of each layer when I am calling the model (see below)</p>
  
  <p>Forward Propagation</p>
  
  <pre><code>def forward_propagation(X, parameters):
      parameters_count = len(parameters) // 2 
      A = X
  
      for layer in range(1, parameters_count):
          W = parameters['W' + str(layer)]
          b = parameters['b' + str(layer)]
  
          Z = tf.add(tf.matmul(W, A), b)
          A = tf.nn.relu(Z)
  
      W = parameters['W' + str(parameters_count)]
      b = parameters['b' + str(parameters_count)]
  
      Z = tf.add(tf.matmul(W, A), b)
  
      return Z
  </code></pre>
  
  <p>Compute the cost (I am using the sigmoid function since we are dealing with binary classification)</p>
  
  <pre><code>def compute_cost(Z, Y):    
      logits = tf.transpose(Z)
      labels = tf.transpose(Y)
  
      cost = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits = logits, labels = labels))
      return cost
  </code></pre>
  
  <p>Putting it together</p>
  
  <pre><code>def model(X_train, y_train, X_valid, y_valid, layer_dimensions, alpha = 0.0001, epochs = 10):
  
      ops.reset_default_graph()
      tf.set_random_seed(seed)
  
      (x_rows, m) = X_train.shape
      y_rows = y_train.shape[0]
  
      costs = []
  
      X = tf.placeholder(tf.float32, shape=(x_rows, None), name="X")
      y = tf.placeholder(tf.float32, shape=(y_rows, None), name="y")
  
      parameters = initialize_parameters(layer_dimensions)
      Z = forward_propagation(X, parameters)
      cost = compute_cost(Z, y)
      optimizer = tf.train.AdamOptimizer(learning_rate = alpha).minimize(cost)
  
      init = tf.global_variables_initializer()
      with tf.Session() as sess:
          sess.run(init)
          for epoch in range(epochs):
              _ , epoch_cost = sess.run([optimizer, cost], feed_dict={X: X_train, y: y_train})
              print ("Cost after epoch %i: %f" % (epoch + 1, epoch_cost))
              costs.append(epoch_cost)
  
          parameters = sess.run(parameters)
  
          correct_predictions = tf.equal(tf.argmax(Z), tf.argmax(y))
          accuracy = tf.reduce_mean(tf.cast(correct_predictions, "float"))
  
          print ("Train Accuracy:", accuracy.eval({X: X_train, y: y_train}))
          print ("Test Accuracy:", accuracy.eval({X: X_valid, y: y_valid}))
  
          return parameters
  </code></pre>
  
  <p>Now when I try to train my model it appears to reaches an optimum from the second epoch and the cost changes very little from that point on</p>
  
  <pre><code>parameters = model(X_train, y_train, X_valid, y_valid, [X_train.shape[0], 50, 25, 10, 5, 1])
  </code></pre>
  
  <p>Cost after epoch 1: 8.758244 <br />
  Cost after epoch 2: 0.693096 <br />
  Cost after epoch 3: 0.692992 <br />
  Cost after epoch 4: 0.692737 <br />
  Cost after epoch 5: 0.697333 <br />
  Cost after epoch 6: 0.693062 <br />
  Cost after epoch 7: 0.693151 <br />
  Cost after epoch 8: 0.693152 <br />
  Cost after epoch 9: 0.693152 <br />
  Cost after epoch 10: 0.693155 <br /></p>
  
  <p>Now for the predictions</p>
  
  <pre><code>def predict(X, parameters):
      parameters_count = len(parameters) // 2 
      params = {}
  
      for layer in range(1, parameters_count + 1):
          params['W' + str(layer)] = tf.convert_to_tensor(parameters['W' + str(layer)])
          params['b' + str(layer)] = tf.convert_to_tensor(parameters['b' + str(layer)])
  
      (x_columns, x_rows) = X.shape
      X_test = tf.placeholder(tf.float32, shape=(x_columns, x_rows))
  
      Z = forward_propagation(X_test, params)
      p = tf.argmax(Z)
  
      sess = tf.Session()
      prediction = sess.run(p, feed_dict = {X_test: X})
  
      return prediction
  </code></pre>
  
  <p>However, this will predict 0 in every case..</p>
  
  <pre><code>predictions = predict(X_valid, parameters)
  predictions
  </code></pre>
  
  <p>array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0....</p>