title: Neural network is not being trained, cross-entropy stays about the same
body: <p>I have written the following multilayer perceptron model in TensorFlow, but it is not training. The accuracy stays around 9%, which is equivalent to random guessing, and cross-entropy stay around 2.56 and does not vary much.</p>
  
  <p>The architecture is as follows:</p>
  
  <pre><code>def create_model(fingerprint_input, model_settings, is_training):
      if is_training:
          dropout_prob = tf.placeholder(tf.float32, name='dropout_prob')
      fingerprint_size = model_settings['fingerprint_size']
      label_count = model_settings['label_count']
      weights_1 = tf.Variable(tf.truncated_normal([fingerprint_size, 128], stddev=0.001))
      weights_2 = tf.Variable(tf.truncated_normal([128, 128], stddev=0.001))
      weights_3 = tf.Variable(tf.truncated_normal([128, 128], stddev=0.001))
      weights_out = tf.Variable(tf.truncated_normal([128, label_count], stddev=0.001))
      bias_1 = tf.Variable(tf.zeros([128]))
      bias_2 = tf.Variable(tf.zeros([128]))
      bias_3 = tf.Variable(tf.zeros([128]))
      bias_out = tf.Variable(tf.zeros([label_count]))
      layer_1 = tf.matmul(fingerprint_input, weights_1) + bias_1
      layer_1 = tf.nn.relu(layer_1)
      layer_2 = tf.matmul(layer_1, weights_2) + bias_2
      layer_2 = tf.nn.relu(layer_2)
      layer_3 = tf.matmul(layer_2, weights_3) + bias_3
      layer_3 = tf.nn.relu(layer_3)
      logits = tf.matmul(layer_3, weights_out) + bias_out
      if is_training:
          return logits, dropout_prob
      else:
          return logits
  </code></pre>
  
  <p>It takes the input size as <code>fingerprint_size</code> and the labels size as well as <code>label_count</code>. It has three hidden layers with 128 neurons each. I'm following the TensorFlow example on a speech data set, which provides a framework for everything else. In the documentation, all I needed to do is to include my own neural network architecture and my method should have those arguments defined and return the logits.</p>
  
  <p>When I trained another predefined architecture, with the same inputs and output, the neural network trains. But this one is not training. Here is one predefined architecture:</p>
  
  <pre><code>def create_single_fc_model(fingerprint_input, model_settings, is_training):
    if is_training:
      dropout_prob = tf.placeholder(tf.float32, name='dropout_prob')
    fingerprint_size = model_settings['fingerprint_size']
    label_count = model_settings['label_count']
    weights = tf.Variable(
        tf.truncated_normal([fingerprint_size, label_count], stddev=0.001))
    bias = tf.Variable(tf.zeros([label_count]))
    logits = tf.matmul(fingerprint_input, weights) + bias
    if is_training:
      return logits, dropout_prob
    else:
  return logits
  </code></pre>
  
  <p>The learning rate are 0.001 for the first 15000 steps and 0.0001 for the last 3000 steps. These are the defaults. I also tried with 0.01 and 0.001, but the same result. I think the problem is somewhere in the above implementation.</p>
  
  <p>Any idea?</p>
  
  <p>Thank you in advance!</p>