def diversity(layer):
  """Encourage diversity between each batch element.

  A neural net feature often responds to multiple things, but naive feature
  visualization often only shows us one. If you optimize a batch of images,
  this objective will encourage them all to be different.

  In particular, it caculuates the correlation matrix of activations at layer
  for each image, and then penalizes cossine similarity between them. This is
  very similar to ideas in style transfer, except we're *penalizing* style
  similarity instead of encouraging it.

  Args:
    layer: layer to evaluate activation correlations on.

  Returns:
    Objective.
  """
  def inner(T):
    layer_t = T(layer)
    batch_n, _, _, channels = layer_t.get_shape().as_list()

    flattened = tf.reshape(layer_t, [batch_n, -1, channels])
    grams = tf.matmul(flattened, flattened, transpose_a=True)
    grams = tf.nn.l2_normalize(grams, axis=[1,2], epsilon=1e-10)

    return sum([ sum([ tf.reduce_sum(grams[i]*grams[j])
                      for j in range(batch_n) if j != i])
                for i in range(batch_n)]) / batch_n
  return inner
Encourage diversity between each batch element.

  A neural net feature often responds to multiple things, but naive feature
  visualization often only shows us one. If you optimize a batch of images,
  this objective will encourage them all to be different.

  In particular, it caculuates the correlation matrix of activations at layer
  for each image, and then penalizes cossine similarity between them. This is
  very similar to ideas in style transfer, except we're *penalizing* style
  similarity instead of encouraging it.

  Args:
    layer: layer to evaluate activation correlations on.

  Returns:
    Objective.