Getting very high values in linear regression
Question: I am trying to make a simple MLP to predict values of a pixel of an image -
[original blog](http://evolvingstuff.blogspot.in/2012/12/generating-mona-lisa-
pixel-by-pixel.html) . Here's my earlier attempt using Keras in python -
[link](https://github.com/goelakash/MonaNet/blob/master/MonaNet.py)

I've tried to do the same in tensorflow, but I am getting very large output
values (~10^12) when they should be less than 1.

Here's my code:

    
    
    import numpy as np
    import cv2
    from random import shuffle
    import tensorflow as tf
    
    '''
    Image preprocessing
    '''
    image_file = cv2.imread("Mona Lisa.jpg")
    
    h = image_file.shape[0]
    w = image_file.shape[1]
    
    preX = []
    preY = []
    
    for i in xrange(h):
        for j in xrange(w):
            preX.append([i,j])
            preY.append(image_file[i,j,:].astype('float32')/255.0)
    
    print preX[:5], preY[:5]
    zipped = [i for i in zip(preX,preY)]
    shuffle(zipped)
    
    X_train = np.array([i for (i,j) in zipped]).astype('float32')
    Y_train = np.array([j for (i,j) in zipped]).astype('float32')
    
    print X_train[:10], Y_train[:10]
    
    '''
    Tensorflow code
    '''
    
    def weight_variable(shape):
      initial = tf.truncated_normal(shape, stddev=0.1)
      return tf.Variable(initial)
    
    def bias_variable(shape):
      initial = tf.constant(0.1, shape=shape)
      return tf.Variable(initial)
    
    x = tf.placeholder(tf.float32, shape=[None,2])
    y = tf.placeholder(tf.float32, shape=[None,3])
    
    
    
    '''
    Layers
    '''
    
    w1 = weight_variable([2,300])
    b1 = bias_variable([300])
    L1 = tf.nn.relu(tf.matmul(X_train,w1)+b1)
    
    w2 = weight_variable([300,3])
    b2 = bias_variable([3])
    y_model = tf.matmul(L1,w2)+b2
    
    
    '''
    Training
    '''
    
    # criterion
    MSE = tf.reduce_mean(tf.square(tf.sub(y,y_model)))
    
    # trainer
    train_op = tf.train.GradientDescentOptimizer(learning_rate = 0.01).minimize(MSE)
    
    nb_epochs = 10
    
    init = tf.initialize_all_variables()
    sess = tf.Session()
    
    sess.run(init)
    cost = 0
    
    for i in range(nb_epochs):
        sess.run(train_op, feed_dict ={x: X_train, y: Y_train})
        cost += sess.run(MSE, feed_dict ={x: X_train, y: Y_train})
    
    cost /= nb_epochs
    print cost
    
    
    '''
    Prediction
    '''
    
    pred = sess.run(y_model,feed_dict = {x:X_train})*255.0
    print pred[:10]
    
    output_image = []
    index = 0
    
    h = image_file.shape[0]
    w = image_file.shape[1]
    
    for i in xrange(h):
        row = []
    
        for j in xrange(w):
            row.append(pred[index])
            index += 1
    
        row = np.array(row)
        output_image.append(row)
    
    output_image = np.array(output_image)
    output_image = output_image.astype('uint8')
    cv2.imwrite('out_mona_300x3_tf.png',output_image)
    


Answer: First of all, I think that instead of running the train_op and then the MSE
you can run both ops in a list and reduce your computational cost
significantly.

    
    
    for i in range(nb_epochs):
    cost += sess.run([MSE, train_op], feed_dict ={x: X_train, y: Y_train})
    

Secondly, I suggest always writing out your cost function so you can see what
is going on during the training phase. Either manually print it out or use
tensorboard to log your cost and plot it (you can find examples on the
official tf page). You can also monitor your weights to see that they aren't
blowing up.

A few things you can try: Reduce learning rate, add regularization to weights.
Check that your training set (pixels) really consist of the values that you
expect them to.

