title: How to deal with overfitting in Tensorflow?
body: <p>I'm currently trying to train a image classification convolutional neural network. I'm using an architecture similar to that in the TensorFlow tutorial. After training, I can get a quite high training accuracy and a very low cross entropy. But the test accuracy is always only a little bit higher than random guessing. The neural network seems to suffer from overfitting. In the training process, I have applied stochastic gradient descent and droupout to try to avoid overfitting. But it just doesn't seem to work. </p>
  
  <p>Here is part of my code.</p>
  
  <pre><code>batch_image = np.ndarray(shape=(100,9216), dtype='float')
  batch_class = np.ndarray(shape=(100,10), dtype='float')
  # first convolutinal layer
  w_conv1 = weight_variable([5, 5, 3, 64])
  b_conv1 = bias_variable([64])
  
  x_image = tf.reshape(x, [-1, 48, 64, 3])
  
  h_conv1 = tf.nn.relu(conv2d(x_image, w_conv1) + b_conv1)
  h_pool1 = max_pool_2x2(h_conv1)
  norm1 = tf.nn.lrn(tf.to_float(h_pool1, name='ToFloat'), 4, bias=1.0, alpha=0.001 / 9.0, beta=0.75)
  
  # second convolutional layer
  w_conv2 = weight_variable([5, 5, 64, 64])
  b_conv2 = bias_variable([64])
  h_conv2 = tf.nn.relu(conv2d(norm1, w_conv2) + b_conv2)
  h_pool2 = max_pool_2x2(h_conv2)
  norm2 = tf.nn.lrn(tf.to_float(h_pool2, name='ToFloat'), 4, bias=1.0, alpha=0.001 / 9.0, beta=0.75)
  
  # densely connected layer
  w_fc1 = weight_variable([12*16*64, 512])
  b_fc1 = bias_variable([512])
  
  h_pool2_flat = tf.reshape(norm2, [-1, 12*16*64])
  h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, w_fc1) + b_fc1)
  
  #densely connected layer
  w_fc2 = weight_variable([512, 256])
  b_fc2 = bias_variable([256])
  h_fc2 = tf.nn.relu(tf.matmul(h_fc1, w_fc2) + b_fc2)
  
  # dropout
  keep_prob = tf.placeholder("float")
  h_fc1_drop = tf.nn.dropout(h_fc2, keep_prob)
  
  # readout layer
  w_fc3 = weight_variable([256, 10])
  b_fc3 = bias_variable([10])
  
  y_prob = tf.nn.softmax(tf.matmul(h_fc1_drop, w_fc3) + b_fc3)
  
  # train and evaluate the model
  
  cross_entropy = -tf.reduce_sum(y_ * tf.log(y_prob + 0.000000001))
  train_step = tf.train.GradientDescentOptimizer(1e-4).minimize(cross_entropy)
  correct_prediction = tf.equal(tf.argmax(y_prob, 1), tf.argmax(y_, 1))
  accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
  sess.run(tf.initialize_all_variables())
  
  for i in range(100):
    rand_idx = np.random.randint(17778, size=(100))
    k = 0
    for j in rand_idx:
      batch_image[k] = images[j]
      batch_class[k] = np.zeros(shape=(10))
      batch_class[k, classes[j, 0]] = 1.0
      k+=1
  
    train_step.run(feed_dict={x:batch_image, y_:batch_class, keep_prob:0.5})
    train_accuracy = accuracy.eval(feed_dict={x:batch_image, y_:batch_class, keep_prob:1.0})
    train_ce = cross_entropy.eval(feed_dict={x:batch_image, y_:batch_class, keep_prob:1.0})
  </code></pre>
  
  <p>I am wondering is there any mistake in my code or do I have to apply any other strategies to get a better test accuracy.</p>
  
  <p>Thank you!</p>