Training a simple net doesn't appear to change values of variables more than once
Question: I'm sure I'm missing something obvious. Here's the tail end of my code:

    
    
    # simple loss function
    loss = tf.reduce_sum(tf.abs(tf.sub(x4, yn)))
    
    train_step = tf.train.GradientDescentOptimizer(0.000001).minimize(loss)
    
    with tf.Session() as sess:
        tf.initialize_all_variables().run()
        print(sess.run([tf.reduce_sum(w1), tf.reduce_sum(b1)]))
        for i in range(5):
            # fill in x1 and yn
            sess.run(train_step, feed_dict={x1: in_images, yn: out_images})
            print(sess.run([tf.reduce_sum(w1), tf.reduce_sum(b1)]))
    

The network descending from the loss function is a simple CNN, with conv2d's
and bias_adds, and elu's. I wanted to take a look at how the weights and
biases for the first layer change. The first print returns the expected values
([ +/- 100 or so, 0]), as w1 was initialized with a random normal and b1
initialized with zeros.

The second print statement gives a different value pair, as expected.

What's not expected is that each time through the loop, the second print
statement prints the same value pair, as though each invocation of train_step
is doing the same thing each time, rather than updating the values of the
Variables in the loss network.

What am I missing here?

Here's a cut and paste of the interesting part of the run:

    
    
    I tensorflow/core/common_runtime/gpu/gpu_device.cc:806] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 970, pci bus id: 0000:01:00.0)
    [-50.281082, 0.0]
    W tensorflow/core/common_runtime/bfc_allocator.cc:213] Ran out of memory trying to allocate 3.98GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
    [112.52832, 0.078026593]
    [112.52832, 0.078026593]
    [112.52832, 0.078026593]
    [112.52832, 0.078026593]
    [112.52832, 0.078026593]
    

I can post the network itself if necessary, but I suspect the problem is my
mental model of how tensorflow updates state.

* * *

Here's the entire python program, with a dummy routine for the image input to
show the issue:

    
    
    import tensorflow as tf
    import numpy as np
    from scipy import misc
    
    H = 128
    W = 128
    
    x1 = tf.placeholder(tf.float32, [None, H, W, 1], "input_image")
    yn = tf.placeholder(tf.float32, [None, H-12, W-12, 1], "test_image")
    
    w1 = tf.Variable(tf.random_normal([7, 7, 1, 64]))   # 7x7, 1 input chan, 64 output chans
    b1 = tf.Variable(tf.constant(0.1, shape=[64]))
    x2 = tf.nn.conv2d(x1, w1, [1,1,1,1], "VALID")
    x2 = tf.nn.bias_add(x2, b1)
    x2 = tf.nn.elu(x2)
    
    w2 = tf.Variable(tf.random_normal([5, 5, 64, 32]))  # 5x5, 64 input 32 output chans
    b2 = tf.Variable(tf.constant(0.1, shape=[32]))
    x3 = tf.nn.conv2d(x2, w2, [1,1,1,1], "VALID")
    x3 = tf.nn.bias_add(x3, b2)
    x3 = tf.nn.elu(x3)
    
    w3 = tf.Variable(tf.random_normal([3, 3, 32, 1]))
    b3 = tf.Variable(tf.constant(0.1, shape=[1]))
    x4 = tf.nn.conv2d(x3, w3, [1,1,1,1], "VALID")
    x4 = tf.nn.bias_add(x4, b3)
    x4 = tf.nn.elu(x4)
    
    loss = tf.reduce_sum(tf.abs(tf.sub(x4, yn)))
    
    train_step = tf.train.GradientDescentOptimizer(0.001).minimize(loss)
    
    # fake for testing
    in_images = np.random.rand(20, 128, 128, 1)
    out_images = np.random.rand(20, 116, 116, 1)
    
    with tf.Session() as sess:
        tf.initialize_all_variables().run()
        print(sess.run([tf.reduce_mean(w1), tf.reduce_mean(b1)]))
        for i in range(5):
            # fill in x1 and yn
            sess.run(train_step, feed_dict={x1: in_images, yn: out_images})
            print(sess.run([tf.reduce_mean(w1), tf.reduce_mean(b1)]))
    

I've looked at a bunch of other training examples and I'm still not seeing
what I am doing wrong. Changing the learning rate will just change the numbers
printed but the behavior remains the same, no apparent change from running the
optimizer.


Answer: The error was in the way I computed my loss function. I just added up all of
the errors across the batch, rather than taking the mean error for each pair
of images. The following loss function

    
    
    # simple loss function
    diff_image = tf.abs(tf.sub(x4,yn))
    # sum over all dimensions except batch dim
    err_sum = tf.reduce_sum(diff_image, [1,2,3])
    #take mean over batch
    loss = tf.reduce_mean(err_sum)
    

actually starts converging with the AdamOptimizer. The
GradientDescentOptimizer still exhibits the "change once only" feature, and
I'll go treat it as a bug and post on github.

