title: TensorFlow: Difference in performance of gather for int32 and int64 indices on GPU
body: <p>I run a small, test to evaluate the performance of several TF operations, and noticed an interesting phenomenon. On multiple machines, I consistently get better performance for operations using <code>int64</code> indices vs <code>int32</code> indices. For example, as below for <code>gather</code>, if <code>index_dtype</code> is <code>np.int32</code>, the operation is 2x slower than for <code>np.int64</code>.</p>
  
  <p>My mini test consists of a graph of <code>gather</code> operations that run sequentially (the tuple ensures that there is a dependency on the output of the previous operation):</p>
  
  <pre><code>indices = np.asarray(indices, dtype=index_dtype)
  params = tf.placeholder(dtype=tf.float32)
  
  op = tf.gather(params, indices)
  for _ in range(num_ops):
      op = tf.gather(tf.tuple([params, op])[0], indices)
  </code></pre>
  
  <p>I run the graph multiple times and averaged the time (ignoring the first run).</p>
  
  <p>Since, <code>int32</code> is the native int type for GPU computations, I would expect the opposite behavior. What could be the cause of such difference in performance?</p>