title: Tensorflow: How to get gradients per instance in a batch?
body: <p>I'm looking at the policy gradients sample in this notebook: <a href="https://github.com/ageron/handson-ml/blob/master/16_reinforcement_learning.ipynb" rel="nofollow noreferrer">https://github.com/ageron/handson-ml/blob/master/16_reinforcement_learning.ipynb</a></p>
  
  <p>The relevant code is here:</p>
  
  <pre><code>X = tf.placeholder(tf.float32, shape=[None, n_inputs])
  
  hidden = tf.layers.dense(X, n_hidden, activation=tf.nn.elu, kernel_initializer=initializer)
  logits = tf.layers.dense(hidden, n_outputs)
  outputs = tf.nn.sigmoid(logits)  # probability of action 0 (left)
  p_left_and_right = tf.concat(axis=1, values=[outputs, 1 - outputs])
  action = tf.multinomial(tf.log(p_left_and_right), num_samples=1)
  
  y = 1. - tf.to_float(action)
  cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(labels=y, logits=logits)
  optimizer = tf.train.AdamOptimizer(learning_rate)
  grads_and_vars = optimizer.compute_gradients(cross_entropy)
  gradients = [grad for grad, variable in grads_and_vars]
  gradient_placeholders = []
  grads_and_vars_feed = []
  for grad, variable in grads_and_vars:
      gradient_placeholder = tf.placeholder(tf.float32, shape=grad.get_shape())
      gradient_placeholders.append(gradient_placeholder)
      grads_and_vars_feed.append((gradient_placeholder, variable))
  training_op = optimizer.apply_gradients(grads_and_vars_feed)
  
  ...
  # Run training over a bunch of instances of inputs
              for step in range(n_max_steps):
                  action_val, gradients_val = sess.run([action, gradients], feed_dict={X: obs.reshape(1, n_inputs)})
  ...
  # Then weight each gradient by the action values, average, and feed them back into training_op to apply_gradients()
  </code></pre>
  
  <p>The above works fine, as each run() returns different gradients.</p>
  
  <p>I'd like to batch all this, and feed an array of inputs into run() instead of one input at a time (my environment is different than the one in the sample, so it makes sense for me to batch, and improve performance). Ie:</p>
  
  <pre><code>action_val, gradients_val = sess.run([action, gradients], feed_dict={X: obs_array})
  </code></pre>
  
  <p>Where obs_array has shape <code>[n_instances, n_inputs]</code>.</p>
  
  <p>The problem is that optimizer.compute_gradients(cross_entropy) seems to return a single gradient, even though cross_entropy is a 1d tensor of shape [None, 1]. action_val does return a 1d tensor of actions, as expected - one action per instance in the batch.</p>
  
  <p>Is there any way for me to get an array of gradients, one per instance in the batch? </p>