title: Tensorflow: binary classification results in all-zero or all-one (network is not training itself)
body: <p>I am trying to compare the results of binary classification
  using logistics regression and deep neural networks.</p>
  
  <p>The dataset I have created was
   1. normalized,
   2. Singular Value Decomposed,
   3. and first few vectors of U matrix from SVD are finally selected.</p>
  
  <p>The results from logistic regression looked just fine.
  (accuracy is over 80 % using 10 vectors from step 3)</p>
  
  <p>Now, I have tried to run Deep neural network using R-tensorflow
  and the results are showing all-zero or all-one classification,
  meaning that the network is not training itself, at all.</p>
  
  <p>I have used
  different optimizers such as AdamOptimizer, GradientDescentOptimizer,
  and different activation function such as softmax, ReLu, sigmoid.</p>
  
  <p>I have tried to modify the dimension of training dataset(42941 x 20, 30, 40, ... , 180), number of nodes in hidden layer(4, 8, 16, ..., 1024), and learning rate (1e-2, 1e-3, ..., 1e-10).</p>
  
  <p>All results are either 0.491124 or 0.508876 which is exactly the ratio of zeros or ones in my training dataset.
  (I have manually selected the number of rows to avoid skewed classes
  as below; [c(1:5193,10255:21575,29877:32909),] ).</p>
  
  <p>Is there error in my code? Or any suggestions?</p>
  
  <p>Thanks, in advance.</p>
  
  <p>My R-tensorflow code is as follows.</p>
  
  <p>(Sorry for using R-Tensorflow, not python)</p>
  
  <p>(you can ignore for loop)</p>
  
  <pre><code>library(tensorflow)
  
  n&lt;-9
  
  for (j in 2:(2*n)) {
    for (k in 1:n) {
      for (m in 1:n) {
        b_dl&lt;-as.matrix(b_11_16_pca$u[c(1:nrow(b_11_15_234_z)),1:(j*10)])
        b_test_dl&lt;-as.matrix(b_11_16_pca$u[c((nrow(b_11_15_234_z)+1):nrow(b_11_16_234_z)),1:(j*10)])
  
        sess &lt;- tf$InteractiveSession()
  
        features = tf$placeholder(tf$float32, shape(NULL, as.integer(j*10)))
        labels = tf$placeholder(tf$float32, shape(NULL, 2L))
        keep_prob = tf$placeholder(tf$float32)
  
        # Layers
        w2 = tf$Variable(tf$random_normal(shape(as.integer(j*10), as.integer(2^(1+k)))))
        b2 = tf$Variable(tf$random_normal(shape(as.integer(2^(1+k)))))
        z2 = tf$add(tf$matmul(features, w2), b2)
        h2 = tf$nn$relu(z2)
  
        w1 = tf$Variable(tf$random_normal(shape(as.integer(2^(1+k)), as.integer(2^(1+k)))))
        b1 = tf$Variable(tf$random_normal(shape(as.integer(2^(1+k)))))
        z1 = tf$add(tf$matmul(h2, w1), b1)
        h1 = tf$nn$relu(z1)
  
        w0 = tf$Variable(tf$random_normal(shape(as.integer(2^(1+k)), 2L)))
        b0 = tf$Variable(tf$random_normal(shape(2L)))
        logits = tf$add(tf$matmul(h1, w0), b0)
        y_pred = tf$nn$relu(logits)
  
  
        cost = tf$reduce_mean(-tf$reduce_sum(y_pred * tf$log(logits), reduction_indices=1L))
        optimizer = tf$train$AdamOptimizer(1/10^m)$minimize(cost)
  
        predict = tf$argmax(y_pred, 1L)
        correct_prediction = tf$equal(predict, tf$argmax(labels, 1L))
        accuracy = tf$reduce_mean(tf$cast(correct_prediction, tf$float32))
  
        sess$run(tf$global_variables_initializer())
  
        for (i in 1:10000) {
          if (i %% 100 == 0) {
            train_accuracy &lt;- accuracy$eval(feed_dict = dict(
              features = b_dl[c(1:5193,10255:21575,29877:32909),], labels = a_y[c(1:5193,10255:21575,29877:32909),], keep_prob = 1.0))
            cat(sprintf("step %d, training accuracy %g\n", i, train_accuracy))
          }
          optimizer$run(feed_dict = dict(
            features = b_dl[c(1:5193,10255:21575,29877:32909),], labels = a_y[c(1:5193,10255:21575,29877:32909),], keep_prob = 0.5))
        }
      print(paste0("j is ",j,", k is ",k," and m is ",m))}
    }
  }
  </code></pre>
  
  <p>Code for Tensorflow is as follows.</p>
  
  <pre><code>import tensorflow as tf
  
  sess = tf.InteractiveSession()
  
  features = tf.placeholder(tf.float32, [None, 20])
  labels = tf.placeholder(tf.float32, [None, 2])
  keep_prob = tf.placeholder(tf.float32)
  
  w2 = tf.Variable(tf.random_normal([20, 32]))
  b2 = tf.Variable(tf.random_normal([32]))
  z2 = tf.add(tf.matmul(features, w2), b2)
  h2 = tf.nn.relu(z2)
  
  w1 = tf.Variable(tf.random_normal([32, 32]))
  b1 = tf.Variable(tf.random_normal([32]))
  z1 = tf.add(tf.matmul(h2, w1), b1)
  h1 = tf.nn.relu(z1)
  
  w0 = tf.Variable(tf.random_normal([32, 2]))
  b0 = tf.Variable(tf.random_normal([2]))
  logits = tf.add(tf.matmul(h1, w0), b0)
  y_pred = tf.nn.relu(logits)
  
  
  cost = tf.reduce_mean(-tf.reduce_sum(labels * tf.log(y_pred), reduction_indices=1))
  optimizer = tf.train.AdamOptimizer(1e-2).minimize(cost)
  
  predict = tf.argmax(y_pred, 1)
  correct_prediction = tf.equal(predict, tf.argmax(labels, 1))
  accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
  
  sess.run(tf.global_variables_initializer())
  
  for i in range(10000): 
    if i % 100 == 0: 
      train_accuracy = accuracy.eval(feed_dict = {
        features: b_dl, labels: a_y, keep_prob: 1.0})
      print("step %d, training accuracy %g\n", i, train_accuracy)
    optimizer.run(feed_dict = {
      features: b_dl, labels: a_y, keep_prob: 0.5})
  </code></pre>