Custom gradient for a chain of ops
Question: I've got a chain of standard TensorFlow operations, and I need to specify a
custom gradient for this chain as a whole.

Say that, in the example below, these operations are grouped in a single
Python function: 'my_op'. What I'm trying to do is to specify a custom
gradient for 'my_op'. I had a look at RegisterGradient, gradient_override_map,
and tf.Graph.create_op, but I couldn't find any simple example about how to
use them to define a custom gradient for a group of ops without rewriting the
full operation chain in C++.

    
    
    import numpy as np
    import tensorflow as tf
    
    n = 2
    m = 3
    x = np.random.normal(size=(1, n))
    A = tf.Variable(tf.truncated_normal(shape=(n, m), dtype=tf.float32))
    b = tf.Variable(tf.zeros(shape=(1, m), dtype=tf.float32))
    
    def my_op(a):
        return tf.add(tf.matmul(a, A), b)
    
    x_placeholder = tf.placeholder(tf.float32,shape=[1, n])
    t = my_op(tf.stop_gradient(x_placeholder))
    
    grad = tf.gradients(t, [A])
    
    
    sess = tf.Session()
    sess.run(tf.initialize_all_variables())
    
    result = sess.run(grad, feed_dict={x_placeholder: x})
    
    print(result)
    
    sess.close()
    


Answer: As far as as I can see, the best way you can define a custom gradient(i.e.
give some modification to the plain gradients) is to add a new custom ops in
tensorflow following
[this](https://www.tensorflow.org/versions/master/how_tos/adding_an_op/index.html#adding-
a-new-op). As you can see, for a custom op outputing the input, you can define
the gradients of it in python by making use of
`@ops.RegisterGradient("MyOp")`.

