title: TensorFlow: Implementing Spearman Distance as the Objective Function
body: <p>In order to make my issue reproducible, I have generated the following <code>.csv</code> file using iris flower data set (10 arbitrary rows, all columns standard normalized) and a minimal neural network model (which predicts petal width using sepal length, sepal width and petal length) by modifying an MNIST example that I found on the internet. Scroll down to see my question!</p>
  
  <blockquote>
    <p>iris.csv</p>
  </blockquote>
  
  <pre><code>"Sepal.Length","Sepal.Width","Petal.Length","Petal.Width","Species"
  0.0551224773430978,-0.380319414627833,-0.335895230408602,-0.548226210538025,"versicolor"
  1.48830688826362,-1.01418510567422,1.37931445678426,0.614677872421422,"virginica"
  0.606347250774068,0.887411967464943,0.450242542888127,0.780807027129915,"virginica"
  -0.606347250774067,-1.64805079672061,0.235841331989019,0.44854871771293,"virginica"
  1.15757202420504,-1.01418510567422,0.950512034986045,0.44854871771293,"virginica"
  -1.92928670700839,0.887411967464943,-2.33697319880027,-2.37564691233144,"setosa"
  0.38585734140168,0.253546276418555,0.307308402288722,1.1130653365469,"virginica"
  -0.826837160146455,0.253546276418555,-0.478829371008007,-0.548226210538025,"versicolor"
  0.0551224773430978,1.52127765851133,-0.192961089809197,-0.21596790112104,"versicolor"
  -0.385857341401679,0.253546276418555,0.021440121089911,0.282419563004437,"virginica"
  </code></pre>
  
  <blockquote>
    <p>nn.py</p>
  </blockquote>
  
  <pre><code>import pandas as pd
  import numpy as np
  import tensorflow as tf
  import scipy.stats
  
  # Import iris data
  data = pd.read_csv("iris.csv")
  input = data[["Sepal.Length", "Sepal.Width", "Petal.Length"]]
  target = data[["Petal.Width"]]
  
  # Parameters
  learning_rate = 0.001
  training_epochs = 6000
  
  # Network Parameters
  n_hidden_1 = 5 # 1st layer number of features
  n_hidden_2 = 5 # 2nd layer number of features
  n_input = 3 # data input
  n_output = 1 # data output
  
  # tf Graph input
  x = tf.placeholder("float", [None, n_input])
  y = tf.placeholder("float", [None, n_output])
  
  # Create model
  def multilayer_network(x, weights, biases):
    # Hidden layer with TanH activation
    layer_1 = tf.add(tf.matmul(x, weights['h1']), biases['b1'])
    layer_1 = tf.tanh(layer_1)
    # Hidden layer with TanH activation
    layer_2 = tf.add(tf.matmul(layer_1, weights['h2']), biases['b2'])
    layer_2 = tf.tanh(layer_2)
    # Output layer with linear activation
    out_layer = tf.matmul(layer_2, weights['out']) + biases['out']
    return out_layer
  
  # Store layers weight &amp; bias
  weights = {
    'h1': tf.Variable(tf.random_normal([n_input, n_hidden_1])),
    'h2': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])),
    'out': tf.Variable(tf.random_normal([n_hidden_2, n_output]))
  }
  biases = {
    'b1': tf.Variable(tf.random_normal([n_hidden_1])),
    'b2': tf.Variable(tf.random_normal([n_hidden_2])),
    'out': tf.Variable(tf.random_normal([n_output]))
  }
  
  # Construct model
  pred = multilayer_network(x, weights, biases)
  
  # Define loss and optimizer
  cost = tf.reduce_mean(tf.square(pred-y))
  optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
  
  # Initializing the variables
  init = tf.initialize_all_variables()
  
  # Launch the graph
  with tf.Session() as sess:
    sess.run(init)
  
    # Training cycle
    for epoch in range(training_epochs):
      # Run optimization op (backprop) and cost op (to get loss value)
      _, c = sess.run([optimizer, cost], feed_dict={x: input, y: target})
  
      # Display logs per epoch step
      if epoch % 1000 == 0:
        print "Epoch:", '%04d' % (epoch+1), "cost=", "{:.9f}".format(c)
  
  print "Optimization Finished!"
  </code></pre>
  
  <p>Here is an example training session result:</p>
  
  <pre><code>$ python nn.py
  Epoch: 0001 cost= 3.000185966
  Epoch: 1001 cost= 0.031734336
  Epoch: 2001 cost= 0.000614795
  Epoch: 3001 cost= 0.000008422
  Epoch: 4001 cost= 0.000000057
  Epoch: 5001 cost= 0.000000000
  Optimization Finished!
  </code></pre>
  
  <hr>
  
  <p>My idea was to replace Mean Square Error with the Spearman distance, which I recently learnt about, as my objective function. Following the definition:</p>
  
  <p><a href="https://i.stack.imgur.com/2Awye.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/2Awye.png" alt="FORMULA"></a></p>
  
  <p>I wrote a function that returns the ranking of a vector:</p>
  
  <pre><code>import scipy.stats
  
  def rank(vector):
    return scipy.stats.rankdata(vector, method="min")
  </code></pre>
  
  <p>Using TensorFlow's method <code>py_func</code>, I defined my cost tensor as follows.</p>
  
  <pre><code>pred = tf.to_float(tf.py_func(rank, [pred], [tf.int64])[0])
  y = tf.to_float(tf.py_func(rank, [y], [tf.int64])[0])
  
  cost = tf.reduce_mean(tf.square(y-pred))
  </code></pre>
  
  <p>However, this gave me the error</p>
  
  <pre><code>ValueError: No gradients provided for any variable: ((None, &lt;tensorflow.python.ops.variables.Variable object at 0x7f67ffe4ee90&gt;), (None, &lt;tensorflow.python.ops.variables.Variable object at 0x7f66ed3c4990&gt;), (None, &lt;tensorflow.python.ops.variables.Variable object at 0x7f66ed357310&gt;), (None, &lt;tensorflow.python.ops.variables.Variable object at 0x7f66ed357190&gt;), (None, &lt;tensorflow.python.ops.variables.Variable object at 0x7f66ed380350&gt;), (None, &lt;tensorflow.python.ops.variables.Variable object at 0x7f66ed3801d0&gt;))
  </code></pre>
  
  <p>I do not understand what the underlying problem is. Any direction you could provide me would be greatly appreciated!</p>