title: Pearson Correlation as a Loss Function
body: <p>I'm training a TF Feed Forward network where my goal is to produce predictions from 0 to 1 that are as close as possible to the target scores. A single training instance consists of around 450 features and there are circa 1500 examples in the dataset. I use 4 layers in my network each with a Relu activation and then the final 'out' layer has a sigmoid activation. When I use MSE as the loss function, I get decent (but not optimal) results. I have tried to use the following as a loss function:</p>
  
  <pre><code># Define loss and optimizer
  #pearson correlation as loss function
  length = 443
  
  #apply regularization (l2)
  Beta = 0.01
  regularizer = tf.nn.l2_loss(weights['h1']) +   
  tf.nn.l2_loss(weights['h2']) + tf.nn.l2_loss(weights['h3']) + 
  tf.nn.l2_loss(weights['h4'])
  
  #used to report correlation 
  pearson = tf.contrib.metrics.streaming_pearson_correlation(intensity, 
  Y, name="pearson")
  
  
  #pearson corr. as loss?
  # multiply by -1 to maximize correlation i.e. minimize negative 
  correlation 
  original_loss = -1 * length * tf.reduce_sum(tf.multiply(intensity, Y)) 
  - (tf.reduce_sum(intensity) * tf.reduce_sum(Y))
  divisor = tf.sqrt(
          (length * tf.reduce_sum(tf.square(intensity)) - 
  tf.square(tf.reduce_sum(intensity)))) *\
          tf.sqrt(
          length * tf.reduce_sum(tf.square(Y)) - 
  tf.square(tf.reduce_sum(Y)))
  
   loss_op = tf.truediv(original_loss, divisor)
   loss_op = tf.reduce_mean(loss_op + Beta * regularizer)
  #Init optimizer
   optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate, 
   epsilon = 1e-09)
   train_op = optimizer.minimize(loss_op) 
  </code></pre>
  
  <p>The idea is to minimise negative correlation, i.e. maximise positive correlation. However, after a lot of experimentation with the hyperparameters, this still gives me 'nan' error and reports 'nan' Pearson correlation. Any ideas as to why that could be?</p>