Convnet training error does not decrease I'm training a convoluted neural net to drive a toy car, and no matter what I do the training accuracy does not increase beyond 30-35%, which is where it starts when the convnet is randomly initialized. What's strange is that a much simpler model, a neural network with a single hidden layer and no convolution, does substantially better, consistently getting accuracy of 65-75%. I've been working on this project for over a year and feel like I've tried everything to make the convnet better. What am I doing wrong?

Notes:


Dataset contains 150,000 records, but since my video feed generates 20 frames per second and since frames don't change much from second to second, it's more like 150,000 / 20 = 7,500 unique records. 
Three equally proportioned classes: turn left, go straight, turn right
Both the convnet and simple NN use Tensorflow's AdamOptimizer. The simple net does well with 1e-5 and 1e-4, but the convnet doesn't do well with any value, I've tried 1e-2 all the way through 1e-6
Simple NN uses the sigmoid activation function, the convnet is all relu activations
Both models read the same data. I have a single sampling and data augmentation class that's shared among all my models, so my data isn't bad because the simple net works fine on the same inputs 
My convnet can't even overfit on the training data, so it's not data size that's the issue, in my opinion
Overfitting doesn't seem to be a problem with either model: training and validation sets tend to have similar performance, give or take 5%
I'm using initial = tf.truncated_normal(shape, stddev=0.1) to initialize all weights
I'm using initial = tf.constant(0.1, shape=shape) to initialize all biases
It could be specifically the convolution that's the problem, since a shallow single-hidden-layer convnet did poorly but a two-hidden-layer fully-connected neural net with no convolution gets around 65% accuracy


Convnet Notes (all yield same poor results):


Batch normalization
Max pooling
50% drop-out probability
Various stride sizes
Various depths of layers: I've tried 2-5 convent layers, multiple 2-4 fully connected layers. I've had as few as 3 layers and as many as 9. Convnet layers have between 32 and 64 neurons, fully connected layers have between 32 and 512 neurons. 
3D convolution (took too much memory and caused my GPU to crash)
1x1 convolutions
The one thing I haven't tried is transfer learning, and I hope to only use that as a last resort since the simple net works fine


Code:


Simple neural net
Deep Convnet
In your convnet code, you compute the cross entropy manually:



cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y_conv), reduction_indices=[1]))
train_step = tf.train.AdamOptimizer(1e-5).minimize(cross_entropy)


If you do that, you may run into numerical stability issues. Instead, you should use

tf.nn.softmax_cross_entropy_with_logits()


If using tensorflow version 1.8 or later, use

tf.nn.softmax_cross_entropy_with_logits_v2()


and weep at the state of tensorflow's API.

See this issue for more information: https://github.com/tensorflow/tensorflow/issues/2462