<p dir="auto">We are getting unexpected names (they lack outer name scope) in <code class="notranslate">NamedOutputs</code> tuples added to collections using <code class="notranslate">tf.contrib.layers</code> <code class="notranslate">outputs_collections</code>. The following code demonstrates the issue:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import tensorflow as tf
from tensorflow.contrib import slim


with tf.name_scope(&quot;train&quot;):
    with slim.arg_scope([slim.fully_connected, slim.flatten],
                        outputs_collections=tf.GraphKeys.ACTIVATIONS):
        ph = tf.placeholder(tf.float32, [2, 2])
        fc = slim.fully_connected(ph, 10)
        flat = slim.flatten(ph)

{print(&quot;name in tuple: &quot;, no.name, &quot;, tensor name:&quot;, no.outputs.name)
 for no in tf.get_collection(tf.GraphKeys.ACTIVATIONS)}"><pre class="notranslate"><code class="notranslate">import tensorflow as tf
from tensorflow.contrib import slim


with tf.name_scope("train"):
    with slim.arg_scope([slim.fully_connected, slim.flatten],
                        outputs_collections=tf.GraphKeys.ACTIVATIONS):
        ph = tf.placeholder(tf.float32, [2, 2])
        fc = slim.fully_connected(ph, 10)
        flat = slim.flatten(ph)

{print("name in tuple: ", no.name, ", tensor name:", no.outputs.name)
 for no in tf.get_collection(tf.GraphKeys.ACTIVATIONS)}
</code></pre></div>
<p dir="auto">The output is:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="name in tuple:  fully_connected , tensor name: train/fully_connected/Relu:0
name in tuple:  train/Flatten , tensor name: train/Flatten/Reshape:0"><pre class="notranslate"><code class="notranslate">name in tuple:  fully_connected , tensor name: train/fully_connected/Relu:0
name in tuple:  train/Flatten , tensor name: train/Flatten/Reshape:0
</code></pre></div>
<p dir="auto">We have tracked the cause of this down in to <code class="notranslate">tf.contrib.layers</code>. For layers that use internal variables (fully_connected, conv2d, ...), final outputs are added to collections based on internal variable_scope name. Please see:</p>
<p dir="auto">tensorflow/tensorflow/contrib/layers/python/layers/layers.py lines 758, 835 (version 0.10, commit <a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/tensorflow/tensorflow/commit/1df3fb0b4ae5915364f09e233496e98a99a4a886/hovercard" href="https://github.com/tensorflow/tensorflow/commit/1df3fb0b4ae5915364f09e233496e98a99a4a886"><tt>1df3fb0</tt></a>)</p>
<p dir="auto">It is our understanding that activation names generally fall under name_scopes, which is consistent with actual op names in the output above.</p>
<p dir="auto">This issue makes it impossible to retrieve items from collections filtered down with a name scope, an approach that we are trying to use for decoupling op creation and summarizing. It seems a valid use-case.</p>
<h1 dir="auto">Problem description</h1>
<p dir="auto">TensorFlow version r0.10</p>
<p dir="auto">It seems to me that there is a bug with the outputs_collections in tensorflow/tensorflow/contrib/layers. When trying to add the outputs of the layers (say, conv2d) to the tf.GraphKeys.ACTIVATIONS collection, I run into errors with NamedOutputs such as the following:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="...
Run summarize_activations() from tensorflow/tensorflow/contrib/layers/python/layers/summaries.py
...
/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/summaries.pyc in summarize_collection(collection, name_filter, summarizer)
    160     if name_filter is None or re.match(name_filter, op.op.name):
    161       tensors.append(op)
--&gt; 162   return summarize_tensors(tensors, summarizer)
    163
    164

/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/summaries.pyc in summarize_tensors(tensors, summarizer)
    150 def summarize_tensors(tensors, summarizer=summarize_tensor):
    151   &quot;&quot;&quot;Summarize a set of tensors.&quot;&quot;&quot;
--&gt; 152   return [summarizer(tensor) for tensor in tensors]
    153
    154

/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/summaries.pyc in summarize_tensor(tensor, tag)
    135   &quot;&quot;&quot;
    136   # Skips string tensors and boolean tensors (not handled by the summaries).
--&gt; 137   if (tensor.dtype.is_compatible_with(dtypes.string) or
    138       tensor.dtype.base_dtype == dtypes.bool):
    139     return None

AttributeError: 'NamedOutputs' object has no attribute 'dtype'"><pre class="notranslate"><code class="notranslate">...
Run summarize_activations() from tensorflow/tensorflow/contrib/layers/python/layers/summaries.py
...
/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/summaries.pyc in summarize_collection(collection, name_filter, summarizer)
    160     if name_filter is None or re.match(name_filter, op.op.name):
    161       tensors.append(op)
--&gt; 162   return summarize_tensors(tensors, summarizer)
    163
    164

/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/summaries.pyc in summarize_tensors(tensors, summarizer)
    150 def summarize_tensors(tensors, summarizer=summarize_tensor):
    151   """Summarize a set of tensors."""
--&gt; 152   return [summarizer(tensor) for tensor in tensors]
    153
    154

/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/summaries.pyc in summarize_tensor(tensor, tag)
    135   """
    136   # Skips string tensors and boolean tensors (not handled by the summaries).
--&gt; 137   if (tensor.dtype.is_compatible_with(dtypes.string) or
    138       tensor.dtype.base_dtype == dtypes.bool):
    139     return None

AttributeError: 'NamedOutputs' object has no attribute 'dtype'
</code></pre></div>
<h1 dir="auto">Root of problem</h1>
<h2 dir="auto">README and code are not consistent</h2>
<p dir="auto">The readme (located at tensorflow/tensorflow/contrib/layers/README.md) indicates that "weights, biases, and activations (i.e., outputs) are, by default, added to the specified collections." They even show a piece of code such as <code class="notranslate">output_collections=(tf.GraphKeys.ACTIVATIONS,)</code>. However, looking into the code in tensorflow/tensorflow/contrib/layers/python/layers/layers.py it seems to me that there is nothing adding activations to the tf.GraphKeys.ACTIVATIONS collection. I believe one solution to this problem is to use the <code class="notranslate">_apply_activation</code> function already written in the file:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="def _apply_activation(y, activation_fn, output_collections):
  if activation_fn:
    y = activation_fn(y)
  ops.add_to_collections(list(output_collections or []) +
                         [ops.GraphKeys.ACTIVATIONS], y)
  return y"><pre class="notranslate"><code class="notranslate">def _apply_activation(y, activation_fn, output_collections):
  if activation_fn:
    y = activation_fn(y)
  ops.add_to_collections(list(output_collections or []) +
                         [ops.GraphKeys.ACTIVATIONS], y)
  return y
</code></pre></div>
<p dir="auto">instead of using only</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="if activation_fn:
      outputs = activation_fn(outputs)
return utils.collect_named_outputs(outputs_collections, sc.name, outputs)"><pre class="notranslate"><code class="notranslate">if activation_fn:
      outputs = activation_fn(outputs)
return utils.collect_named_outputs(outputs_collections, sc.name, outputs)
</code></pre></div>
<h2 dir="auto">Code walkthrough</h2>
<p dir="auto">tensorflow/tensorflow/contrib/layers/python/layers/layers.py uses this line several times (such as in conv2d):<br>
<code class="notranslate">utils.collect_named_outputs(outputs_collections, sc, outputs)</code></p>
<p dir="auto">Opening tensorflow/tensorflow/contrib/layers/python/layers/utils.py we see that this ultimately calls:<br>
<code class="notranslate">ops.add_to_collections(collections, NamedOutputs(name, outputs))</code></p>
<p dir="auto">However, it seems to me that when using the summarize_activations function in <a href="https://github.com/tensorflow/tensorflow/blob/r0.10/tensorflow/contrib/layers/python/layers/summaries.py">https://github.com/tensorflow/tensorflow/blob/r0.10/tensorflow/contrib/layers/python/layers/summaries.py</a>, it expects the value to be <code class="notranslate">outputs</code> rather than <code class="notranslate">NamedOutputs(name, outputs)</code>. In other words, the summarize_activations function seems to expect something along the lines of:<br>
<code class="notranslate">ops.add_to_collections(collections, outputs)</code><br>
rather than<br>
<code class="notranslate">ops.add_to_collections(collections, NamedOutputs(name, outputs))</code></p>
<h3 dir="auto">Further description (not particularly informative if you are already familiar with the tensorflow repo)</h3>
<p dir="auto">If we look into what <code class="notranslate">ops.add_to_collections</code> is doing, we see that in tensorflow/tensorflow/python/framework/ops.py this ultimately calls:<br>
<code class="notranslate">self.add_to_collection(name, value)</code></p>
<p dir="auto"><code class="notranslate">add_to_collection(...)</code> is described in the same file as:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="  def add_to_collection(self, name, value):
    &quot;&quot;&quot;Stores `value` in the collection with the given `name`.
    Note that collections are not sets, so it is possible to add a value to
    a collection several times.
    Args:
      name: The key for the collection. The `GraphKeys` class
        contains many standard names for collections.
      value: The value to add to the collection."><pre class="notranslate"><code class="notranslate">  def add_to_collection(self, name, value):
    """Stores `value` in the collection with the given `name`.
    Note that collections are not sets, so it is possible to add a value to
    a collection several times.
    Args:
      name: The key for the collection. The `GraphKeys` class
        contains many standard names for collections.
      value: The value to add to the collection.
</code></pre></div>
<p dir="auto">So, our <code class="notranslate">value</code> here is <code class="notranslate">value = NamedOutputs(name, outputs)</code> from earlier.</p>
<p dir="auto">However, if we look at the summarize_activations in tensorflow/tensorflow/contrib/layers/python/layers/summaries.py, we see that:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="def summarize_activations(name_filter=None, summarizer=summarize_activation):
  &quot;&quot;&quot;Summarize activations, using `summarize_activation` to summarize.&quot;&quot;&quot;
  return summarize_collection(ops.GraphKeys.ACTIVATIONS, name_filter,
                              summarizer)

def summarize_collection(collection, name_filter=None,
                         summarizer=summarize_tensor):
  &quot;&quot;&quot;Summarize a graph collection of tensors, possibly filtered by name.&quot;&quot;&quot;
  tensors = []
  for op in ops.get_collection(collection):
    if name_filter is None or re.match(name_filter, op.op.name):
      tensors.append(op)
  return summarize_tensors(tensors, summarizer)"><pre class="notranslate"><code class="notranslate">def summarize_activations(name_filter=None, summarizer=summarize_activation):
  """Summarize activations, using `summarize_activation` to summarize."""
  return summarize_collection(ops.GraphKeys.ACTIVATIONS, name_filter,
                              summarizer)

def summarize_collection(collection, name_filter=None,
                         summarizer=summarize_tensor):
  """Summarize a graph collection of tensors, possibly filtered by name."""
  tensors = []
  for op in ops.get_collection(collection):
    if name_filter is None or re.match(name_filter, op.op.name):
      tensors.append(op)
  return summarize_tensors(tensors, summarizer)
</code></pre></div>
<p dir="auto">In other words, <code class="notranslate">summarize_collection(...)</code> expects the collection to contain tensors instead of NamedOutputs(name, outputs).</p>