title: Autoencoder loss is not decreasing (and starts very high)
body: <p>I have the following function which is supposed to autoencode my data.</p>
  
  <p>My data can be thought of as an image of length 100, width 2, and it has 2 channels (100, 2, 2)</p>
  
  <pre><code>def construct_ae(input_shape):
      encoder_input = tf.placeholder(tf.float32, input_shape, name='x')
      with tf.variable_scope("encoder"):
          flattened = tf.layers.flatten(encoder_input)
          e_fc_1 = tf.layers.dense(flattened, units=150, activation=tf.nn.relu)
          encoded = tf.layers.dense(e_fc_1, units=75, activation=None)
  
      with tf.variable_scope("decoder"):
          d_fc_1 = tf.layers.dense(encoded, 150, activation=tf.nn.relu)
          d_fc_2 = tf.layers.dense(d_fc_1, 400, activation=None)
          decoded = tf.reshape(d_fc_2, input_shape)
  
      with tf.variable_scope('training'):
          loss = tf.losses.mean_squared_error(labels=encoder_input, predictions=decoded)
          cost = tf.reduce_mean(loss)
  
          optimizer = tf.train.AdamOptimizer(learning_rate=0.001).minimize(cost)
          return optimizer
  </code></pre>
  
  <p>I'm running into the issue where my cost is on the order of 1.1e9, and it's not decreasing over time</p>
  
  <p><a href="https://i.stack.imgur.com/QdkxA.png" rel="noreferrer"><img src="https://i.stack.imgur.com/QdkxA.png" alt="cost over time"></a></p>
  
  <p>I visualized the gradients (removed the code because it would just clutter things) and I think something is wrong there? But I'm not sure</p>
  
  <p><a href="https://i.stack.imgur.com/odCfC.png" rel="noreferrer"><img src="https://i.stack.imgur.com/odCfC.png" alt="enter image description here"></a></p>
  
  <p><strong>Questions</strong></p>
  
  <p>1) Does anything in the construction of the network look incorrect? </p>
  
  <p>2) Does the data need to be normalized between 0-1?</p>
  
  <p>3) I hit NaNs sometimes when I try increasing the learning rate to 1. Is that indicative of anything? </p>
  
  <p>4) I think I should probably use a CNN but I ran into the same issues so I thought I'd move to an FC since it's likely easier to debug. </p>
  
  <p>5) I imagine I'm using the wrong loss function but I can't really find any papers regarding the right loss to use. If anyone can direct me to one I'd be very appreciative</p>