title: CNN in Tensorflow - loss remains constant
body: <p>I am just at the beginning of my career in machine learning and wanted to create simple CNN to classify 2 different kind of leaves (belonging to 2 different species of trees). Before gathering huge amount of pictures of leaves, I decided to create very small, simple CNN in Tensorflow and train it on only one image, to check, wheter the code is ok. I normalized the photo of size 256x256(x 3 channels) to &lt;0,1> and created 4 layers (2 conv and 2 dense) network. Unfortunately, loss almost always tended to some constant value (usually some integer) from the very beginning. I thought something was wrong with the picture, so I replaced it with a random numpy array of the same dimensions. Unfortunately, loss was still constant. Sometimes the net seemed to learn, because loss was decreasing, but most times was constant from the very beginning. Could anyone be of help explaining, why is that so? I read that training with one example is the best way to check wheter your code lacks bugs, but the longer I struggle with it, the less can I see. </p>
  
  <p>Here is my code (based on this TensorFlow tutorial <a href="https://www.tensorflow.org/versions/r0.12/tutorials/mnist/pros/" rel="nofollow noreferrer">1</a>). I used exponential linear units, because I thought that my problems were caused by 0 gradient in badly initialized ReLUs.</p>
  
  <pre><code>import matplotlib.pyplot as plt
  import numpy as np
  from numpy import random
  from sklearn import utils
  import tensorflow as tf
  
  #original dataset of 6 leaves
  # input = [ndimage.imread("E:\leaves\dab1.jpg"),
  #         ndimage.imread("E:\leaves\dab2.jpg"),
  #        ndimage.imread("E:\leaves\dab3.jpg"),
  #        ndimage.imread("E:\leaves\klon1.jpg"),
  #        ndimage.imread("E:\leaves\klon2.jpg"),
  #        ndimage.imread("E:\leaves\klon3.jpg")]
  
  #normalize each image (originally uint8)
  #input=[input/255 for i in range(len(input))
  
  #temporary testing dataset, mimicking 6 images, each 3-channel, of dimension 256x256
  input=[random.randn(256,256,3)]
         # random.randn(256, 256, 3),
         # random.randn(256, 256, 3),
         # random.randn(256, 256, 3),
         # random.randn(256, 256, 3),
         # random.randn(256, 256, 3)]
  
  #each image belong to one of two classes
  labels=[[1]]#,[1,0],[1,0],[0,1],[0,1],[0,1]]
  
  
  def weight_variable(shape):
    initial = tf.truncated_normal(shape, stddev=.1)
    return tf.Variable(initial)
  
  def bias_variable(shape):
    initial = tf.truncated_normal(shape, stddev=.1)
    return tf.Variable(initial)
  
  def conv2d(x, W):
    return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')
  
  x = tf.placeholder(tf.float32, shape=[None, 256,256,3])
  y_ = tf.placeholder(tf.float32, shape=[None, 1])
  
  x_image = tf.reshape(x, [-1,256,256,3])
  
  #first conv layer
  W_conv1 = weight_variable([5,5, 3,8])
  b_conv1 = bias_variable([8])
  h_conv1 = tf.nn.elu(conv2d(x_image, W_conv1) + b_conv1)
  
  #second conv layer
  W_conv2 = weight_variable([5,5, 8,16])
  b_conv2 = bias_variable([16])
  h_conv2 = tf.nn.elu(conv2d(h_conv1, W_conv2) + b_conv2)
  
  #first dense layer
  W_fc1 = weight_variable([256*256*16, 10])
  b_fc1 = bias_variable([10])
  out_flat = tf.reshape(h_conv2, [-1, 256*256*16])
  h_fc1 = tf.nn.elu(tf.matmul(out_flat, W_fc1) + b_fc1)
  
  #second dense layer
  W_fc2 = weight_variable([10, 1])
  b_fc2 = bias_variable([1])
  h_fc2 = tf.nn.elu(tf.matmul(h_fc1, W_fc2) + b_fc2)
  
  #tried also with softmax with logits
  cross_entropy=tf.losses.mean_squared_error(predictions=h_fc2, labels=y_)
  train_step = tf.train.AdamOptimizer(1e-3).minimize(cross_entropy)
  
  print("h2", h_fc2.shape)
  print("y", y_.shape)
  
  sess=tf.Session()
  sess.run(tf.global_variables_initializer())
  loss = []
  for i in range(10):
      sess.run(train_step, feed_dict={x:input, y_:labels})
      input, labels = utils.shuffle(input, labels)
      loss.append(sess.run(cross_entropy, feed_dict={x:input, y_:labels}))
      print(i, " LOSS: ", loss[-1])
  
  np.set_printoptions(precision=3, suppress=True)
  for i in range(len(input)):
      print(labels[i], sess.run(h_fc2, feed_dict={x:[input[i]], y_:[labels[i]]}))
  
  plt.plot(loss)
  plt.show()
  </code></pre>
  
  <p>And here a list of what I tried:</p>
  
  <ol>
  <li>The base code above results in loss almost always equal to exactly 4.0</li>
  <li>Expanded training time to 100 epochs. It turned out, that the probability of achieving constant loss increased. It is strange, because in my opinion the number of epochs should change anything in the early stage of training.</li>
  <li>I changed the number of feature maps to 32 in I layer, 64 in II layer and to 100 neurons in dense layer</li>
  <li>Because my output is binary, originally I used only single output. I changed it to excluding 2 outputs. It changed loss to 2.5. It turned out, that my outputs tended to be [-1,-1], while label was [1,0]</li>
  <li>I tried various learning rates, from 0.001 to 0.00005</li>
  <li>I initialized weights and biases with standard deviation equal to 2 instead of 0.1. Loss seemed to decrease, but achieved high values, like 1e10. So I changed the number of epochs from 10 to 100.. and again, loss is 2.5 from the very beginning. After returning to 10 epochs, loss remained 2.5</li>
  <li>I expanded dataset to 6 elements. Loss is the same as previously. </li>
  </ol>
  
  <p>Does anyone has any idea, why does this happen? As far as I know, if the net couldn't generalize, the loss would't decrease and rather increase/oscillate but not remain constant?</p>