<p dir="auto"><em>Please make sure that this is a bug. As per our <a href="https://github.com/tensorflow/tensorflow/blob/master/ISSUES.md">GitHub Policy</a>, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template</em></p>
<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux version 3.10.0-957.10.1.el7.x86_64 (<a href="mailto:mockbuild@x86-040.build.eng.bos.redhat.com">mockbuild@x86-040.build.eng.bos.redhat.com</a>) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) )</li>
<li>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A</li>
<li>TensorFlow installed from (source or binary): binary pip install</li>
<li>TensorFlow version (use command below): v1.13.1-0-g6612da8951 1.13.1</li>
<li>Python version: 3.6.4</li>
<li>Bazel version (if compiling from source): N/A</li>
<li>GCC/Compiler version (if compiling from source): N/A</li>
<li>CUDA/cuDNN version: N/A (CPU)</li>
<li>GPU model and memory: N/A (CPU)</li>
</ul>
<p dir="auto"><strong>Describe the current behavior</strong><br>
Error occurs when trying to compute the Jacobian in the following code</p>
<p dir="auto"><strong>Describe the expected behavior</strong><br>
&lt;tf.Tensor: id=XXX, shape=(2, 1), dtype=float32, numpy=array([[1.], [1.]], dtype=float32)&gt;</p>
<p dir="auto"><strong>Code to reproduce the issue</strong><br>
Provide a reproducible test case that is the bare minimum necessary to generate the problem.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import tensorflow as tf
tf.enable_eager_execution()

with tf.GradientTape() as g:
    x  = tf.constant([1.0])
    g.watch(x)
    y = tf.concat([x,x], axis=0)

g.jacobian(y, x)"><pre class="notranslate"><code class="notranslate">import tensorflow as tf
tf.enable_eager_execution()

with tf.GradientTape() as g:
    x  = tf.constant([1.0])
    g.watch(x)
    y = tf.concat([x,x], axis=0)

g.jacobian(y, x)
</code></pre></div>
<p dir="auto"><strong>Other info / logs</strong><br>
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.</p>
<p dir="auto">Additional info: the expected behavior occurs when creating the <code class="notranslate">GradientTape</code> with persistent=True and replacing the last line of code with <code class="notranslate">g.jacobian(y, x, experimental_use_pfor=False)</code>. However, that workaround is extremely slow with real data.</p>
<p dir="auto">Log of the code that reproduces the issue:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="AttributeError Traceback (most recent call last)
in ()
9 # y = tf.concat([y,y], axis=0)
10
---&gt; 11 g.jacobian(y, x)

/opt/conda/lib/python3.6/site-packages/tensorflow/python/eager/backprop.py in jacobian(self, target, sources, unconnected_gradients, parallel_iterations, experimental_use_pfor)
1021 try:
1022 output = pfor_ops.pfor(loop_fn, target_size,
-&gt; 1023 parallel_iterations=parallel_iterations)
1024 except ValueError as err:
1025 six.reraise(

/opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/parallel_for/control_flow_ops.py in pfor(loop_fn, iters, parallel_iterations)
149 if context.executing_eagerly():
150 f = function.defun(f)
--&gt; 151 return f()
152
153

/opt/conda/lib/python3.6/site-packages/tensorflow/python/eager/function.py in call(self, *args, **kwargs)
862 def call(self, *args, **kwargs):
863 &quot;&quot;&quot;Calls a graph function specialized to the inputs.&quot;&quot;&quot;
--&gt; 864 graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
865 return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access
866

/opt/conda/lib/python3.6/site-packages/tensorflow/python/eager/function.py in _maybe_define_function(self, args, kwargs)
1174 self._input_signature,
1175 autograph=self._autograph,
-&gt; 1176 arg_names=arg_names),
1177 self._function_attributes)
1178 if self._input_signature:

/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, add_control_dependencies, arg_names, op_return_value)
446 tf_decorator.rewrap(python_func, original_func, converted_func)
447
--&gt; 448 func_outputs = python_func(*func_args, **func_kwargs)
449
450 # invariant: func_outputs contains only Tensors, IndexedSlices,

/opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/parallel_for/control_flow_ops.py in f()
146 &quot;&quot;&quot;
147 def f():
--&gt; 148 return _pfor_impl(loop_fn, iters, parallel_iterations=parallel_iterations)
149 if context.executing_eagerly():
150 f = function.defun(f)

/opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/parallel_for/control_flow_ops.py in _pfor_impl(loop_fn, iters, parallel_iterations)
157 with ops.name_scope(&quot;loop_body&quot;):
158 loop_var = array_ops.placeholder(dtypes.int32, shape=[])
--&gt; 159 loop_fn_outputs = loop_fn(loop_var)
160 new_ops = set(ops.get_default_graph().get_operations()) - existing_ops
161 iters = ops.convert_to_tensor(iters)

/opt/conda/lib/python3.6/site-packages/tensorflow/python/eager/backprop.py in loop_fn(i)
1011 self._pop_tape()
1012 return self.gradient(y, flat_sources,
-&gt; 1013 unconnected_gradients=unconnected_gradients)
1014
1015 try:

/opt/conda/lib/python3.6/site-packages/tensorflow/python/eager/backprop.py in gradient(self, target, sources, output_gradients, unconnected_gradients)
944 flat_sources,
945 output_gradients=output_gradients,
--&gt; 946 unconnected_gradients=unconnected_gradients)
947
948 if not self._persistent:

/opt/conda/lib/python3.6/site-packages/tensorflow/python/eager/imperative_grad.py in imperative_grad(tape, target, sources, output_gradients, unconnected_gradients)
70 sources,
71 output_gradients,
---&gt; 72 compat.as_str(unconnected_gradients.value))

/opt/conda/lib/python3.6/site-packages/tensorflow/python/eager/backprop.py in _gradient_function(op_name, attr_tuple, num_inputs, inputs, outputs, out_grads)
129 return [None] * num_inputs
130
--&gt; 131 return grad_fn(mock_op, *out_grads)
132
133

/opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/array_grad.py in _ConcatGradV2(op, grad)
220 def _ConcatGradV2(op, grad):
221 return _ConcatGradHelper(
--&gt; 222 op, grad, start_value_index=0, end_value_index=-1, dim_index=-1)
223
224

/opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/array_grad.py in _ConcatGradHelper(op, grad, start_value_index, end_value_index, dim_index)
115 out_grads = array_ops.split(grad, sizes, non_neg_concat_dim)
116 else:
--&gt; 117 if constant_op.is_constant(concat_dim):
118 # If concat_dim is a constant defined in a different context,
119 # then we duplicate it in the current context to avoid passing it

/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py in is_constant(tensor_or_op)
293 def is_constant(tensor_or_op):
294 if isinstance(tensor_or_op, ops.Tensor):
--&gt; 295 op = tensor_or_op.op
296 else:
297 op = tensor_or_op

/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in op(self)
925 def op(self):
926 raise AttributeError(
--&gt; 927 &quot;Tensor.op is meaningless when eager execution is enabled.&quot;)
928
929 @Property

AttributeError: Tensor.op is meaningless when eager execution is enabled."><pre class="notranslate"><span class="pl-v">AttributeError</span> <span class="pl-v">Traceback</span> (<span class="pl-s1">most</span> <span class="pl-s1">recent</span> <span class="pl-s1">call</span> <span class="pl-s1">last</span>)
<span class="pl-en">in</span> ()
<span class="pl-c1">9</span> <span class="pl-c"># y = tf.concat([y,y], axis=0)</span>
<span class="pl-c1">10</span>
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">11</span> <span class="pl-s1">g</span>.<span class="pl-en">jacobian</span>(<span class="pl-s1">y</span>, <span class="pl-s1">x</span>)

<span class="pl-c1">/</span><span class="pl-s1">opt</span><span class="pl-c1">/</span><span class="pl-s1">conda</span><span class="pl-c1">/</span><span class="pl-s1">lib</span><span class="pl-c1">/</span><span class="pl-s1">python3</span>.<span class="pl-c1">6</span><span class="pl-c1">/</span><span class="pl-s1">site</span><span class="pl-c1">-</span><span class="pl-s1">packages</span><span class="pl-c1">/</span><span class="pl-s1">tensorflow</span><span class="pl-c1">/</span><span class="pl-s1">python</span><span class="pl-c1">/</span><span class="pl-s1">eager</span><span class="pl-c1">/</span><span class="pl-s1">backprop</span>.<span class="pl-s1">py</span> <span class="pl-c1">in</span> <span class="pl-en">jacobian</span>(<span class="pl-s1">self</span>, <span class="pl-s1">target</span>, <span class="pl-s1">sources</span>, <span class="pl-s1">unconnected_gradients</span>, <span class="pl-s1">parallel_iterations</span>, <span class="pl-s1">experimental_use_pfor</span>)
<span class="pl-c1">1021</span> <span class="pl-k">try</span>:
<span class="pl-c1">1022</span> <span class="pl-s1">output</span> <span class="pl-c1">=</span> <span class="pl-s1">pfor_ops</span>.<span class="pl-en">pfor</span>(<span class="pl-s1">loop_fn</span>, <span class="pl-s1">target_size</span>,
<span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">1023</span> <span class="pl-s1">parallel_iterations</span><span class="pl-c1">=</span><span class="pl-s1">parallel_iterations</span>)
<span class="pl-c1">1024</span> <span class="pl-k">except</span> <span class="pl-v">ValueError</span> <span class="pl-k">as</span> <span class="pl-s1">err</span>:
<span class="pl-c1">1025</span> <span class="pl-s1">six</span>.<span class="pl-s1">reraise</span>(

<span class="pl-c1">/</span><span class="pl-s1">opt</span><span class="pl-c1">/</span><span class="pl-s1">conda</span><span class="pl-c1">/</span><span class="pl-s1">lib</span><span class="pl-c1">/</span><span class="pl-s1">python3</span>.<span class="pl-c1">6</span><span class="pl-c1">/</span><span class="pl-s1">site</span><span class="pl-c1">-</span><span class="pl-s1">packages</span><span class="pl-c1">/</span><span class="pl-s1">tensorflow</span><span class="pl-c1">/</span><span class="pl-s1">python</span><span class="pl-c1">/</span><span class="pl-s1">ops</span><span class="pl-c1">/</span><span class="pl-s1">parallel_for</span><span class="pl-c1">/</span><span class="pl-s1">control_flow_ops</span>.<span class="pl-s1">py</span> <span class="pl-c1">in</span> <span class="pl-en">pfor</span>(<span class="pl-s1">loop_fn</span>, <span class="pl-s1">iters</span>, <span class="pl-s1">parallel_iterations</span>)
<span class="pl-c1">149</span> <span class="pl-k">if</span> <span class="pl-s1">context</span>.<span class="pl-en">executing_eagerly</span>():
<span class="pl-c1">150</span> <span class="pl-s1">f</span> <span class="pl-c1">=</span> <span class="pl-s1">function</span>.<span class="pl-en">defun</span>(<span class="pl-s1">f</span>)
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">151</span> <span class="pl-s1">return</span> <span class="pl-en">f</span>()
<span class="pl-c1">152</span>
<span class="pl-c1">153</span>

<span class="pl-c1">/</span><span class="pl-s1">opt</span><span class="pl-c1">/</span><span class="pl-s1">conda</span><span class="pl-c1">/</span><span class="pl-s1">lib</span><span class="pl-c1">/</span><span class="pl-s1">python3</span>.<span class="pl-c1">6</span><span class="pl-c1">/</span><span class="pl-s1">site</span><span class="pl-c1">-</span><span class="pl-s1">packages</span><span class="pl-c1">/</span><span class="pl-s1">tensorflow</span><span class="pl-c1">/</span><span class="pl-s1">python</span><span class="pl-c1">/</span><span class="pl-s1">eager</span><span class="pl-c1">/</span><span class="pl-s1">function</span>.<span class="pl-s1">py</span> <span class="pl-c1">in</span> <span class="pl-en">call</span>(<span class="pl-s1">self</span>, <span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
<span class="pl-c1">862</span> <span class="pl-k">def</span> <span class="pl-s1">call</span>(<span class="pl-s1">self</span>, <span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>):
<span class="pl-c1">863</span> <span class="pl-s">"""Calls a graph function specialized to the inputs."""</span>
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">864</span> <span class="pl-s1">graph_function</span>, <span class="pl-s1">args</span>, <span class="pl-s1">kwargs</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">_maybe_define_function</span>(<span class="pl-s1">args</span>, <span class="pl-s1">kwargs</span>)
<span class="pl-c1">865</span> <span class="pl-k">return</span> <span class="pl-s1">graph_function</span>.<span class="pl-en">_filtered_call</span>(<span class="pl-s1">args</span>, <span class="pl-s1">kwargs</span>) <span class="pl-c"># pylint: disable=protected-access</span>
<span class="pl-c1">866</span>

<span class="pl-c1">/</span><span class="pl-s1">opt</span><span class="pl-c1">/</span><span class="pl-s1">conda</span><span class="pl-c1">/</span><span class="pl-s1">lib</span><span class="pl-c1">/</span><span class="pl-s1">python3</span>.<span class="pl-c1">6</span><span class="pl-c1">/</span><span class="pl-s1">site</span><span class="pl-c1">-</span><span class="pl-s1">packages</span><span class="pl-c1">/</span><span class="pl-s1">tensorflow</span><span class="pl-c1">/</span><span class="pl-s1">python</span><span class="pl-c1">/</span><span class="pl-s1">eager</span><span class="pl-c1">/</span><span class="pl-s1">function</span>.<span class="pl-s1">py</span> <span class="pl-c1">in</span> <span class="pl-en">_maybe_define_function</span>(<span class="pl-s1">self</span>, <span class="pl-s1">args</span>, <span class="pl-s1">kwargs</span>)
<span class="pl-c1">1174</span> <span class="pl-s1">self</span>.<span class="pl-s1">_input_signature</span>,
<span class="pl-c1">1175</span> <span class="pl-s1">autograph</span><span class="pl-c1">=</span><span class="pl-s1">self</span>.<span class="pl-s1">_autograph</span>,
<span class="pl-c1">-&gt;</span> <span class="pl-c1">1176</span> <span class="pl-s1">arg_names</span><span class="pl-c1">=</span><span class="pl-s1">arg_names</span>),
<span class="pl-c1">1177</span> <span class="pl-s1">self</span>.<span class="pl-s1">_function_attributes</span>)
<span class="pl-c1">1178</span> <span class="pl-k">if</span> <span class="pl-s1">self</span>.<span class="pl-s1">_input_signature</span>:

<span class="pl-c1">/</span><span class="pl-s1">opt</span><span class="pl-c1">/</span><span class="pl-s1">conda</span><span class="pl-c1">/</span><span class="pl-s1">lib</span><span class="pl-c1">/</span><span class="pl-s1">python3</span>.<span class="pl-c1">6</span><span class="pl-c1">/</span><span class="pl-s1">site</span><span class="pl-c1">-</span><span class="pl-s1">packages</span><span class="pl-c1">/</span><span class="pl-s1">tensorflow</span><span class="pl-c1">/</span><span class="pl-s1">python</span><span class="pl-c1">/</span><span class="pl-s1">framework</span><span class="pl-c1">/</span><span class="pl-s1">func_graph</span>.<span class="pl-s1">py</span> <span class="pl-c1">in</span> <span class="pl-en">func_graph_from_py_func</span>(<span class="pl-s1">name</span>, <span class="pl-s1">python_func</span>, <span class="pl-s1">args</span>, <span class="pl-s1">kwargs</span>, <span class="pl-s1">signature</span>, <span class="pl-s1">func_graph</span>, <span class="pl-s1">autograph</span>, <span class="pl-s1">add_control_dependencies</span>, <span class="pl-s1">arg_names</span>, <span class="pl-s1">op_return_value</span>)
<span class="pl-c1">446</span> <span class="pl-s1">tf_decorator</span>.<span class="pl-en">rewrap</span>(<span class="pl-s1">python_func</span>, <span class="pl-s1">original_func</span>, <span class="pl-s1">converted_func</span>)
<span class="pl-c1">447</span>
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">448</span> <span class="pl-s1">func_outputs</span> <span class="pl-c1">=</span> <span class="pl-en">python_func</span>(<span class="pl-c1">*</span><span class="pl-s1">func_args</span>, <span class="pl-c1">**</span><span class="pl-s1">func_kwargs</span>)
<span class="pl-c1">449</span>
<span class="pl-c1">450</span> <span class="pl-c"># invariant: func_outputs contains only Tensors, IndexedSlices,</span>

<span class="pl-c1">/</span><span class="pl-s1">opt</span><span class="pl-c1">/</span><span class="pl-s1">conda</span><span class="pl-c1">/</span><span class="pl-s1">lib</span><span class="pl-c1">/</span><span class="pl-s1">python3</span>.<span class="pl-c1">6</span><span class="pl-c1">/</span><span class="pl-s1">site</span><span class="pl-c1">-</span><span class="pl-s1">packages</span><span class="pl-c1">/</span><span class="pl-s1">tensorflow</span><span class="pl-c1">/</span><span class="pl-s1">python</span><span class="pl-c1">/</span><span class="pl-s1">ops</span><span class="pl-c1">/</span><span class="pl-s1">parallel_for</span><span class="pl-c1">/</span><span class="pl-s1">control_flow_ops</span>.<span class="pl-s1">py</span> <span class="pl-c1">in</span> <span class="pl-en">f</span>()
<span class="pl-c1">146</span> """
147 def f():
--&gt; 148 return _pfor_impl(loop_fn, iters, parallel_iterations=parallel_iterations)
149 if context.executing_eagerly():
150 f = function.defun(f)

/opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/parallel_for/control_flow_ops.py in _pfor_impl(loop_fn, iters, parallel_iterations)
157 with ops.name_scope("loop_body"):
158 loop_var = array_ops.placeholder(dtypes.int32, shape=[])
--&gt; 159 loop_fn_outputs = loop_fn(loop_var)
160 new_ops = set(ops.get_default_graph().get_operations()) - existing_ops
161 iters = ops.convert_to_tensor(iters)

/opt/conda/lib/python3.6/site-packages/tensorflow/python/eager/backprop.py in loop_fn(i)
1011 self._pop_tape()
1012 return self.gradient(y, flat_sources,
-&gt; 1013 unconnected_gradients=unconnected_gradients)
1014
1015 try:

/opt/conda/lib/python3.6/site-packages/tensorflow/python/eager/backprop.py in gradient(self, target, sources, output_gradients, unconnected_gradients)
944 flat_sources,
945 output_gradients=output_gradients,
--&gt; 946 unconnected_gradients=unconnected_gradients)
947
948 if not self._persistent:

/opt/conda/lib/python3.6/site-packages/tensorflow/python/eager/imperative_grad.py in imperative_grad(tape, target, sources, output_gradients, unconnected_gradients)
70 sources,
71 output_gradients,
---&gt; 72 compat.as_str(unconnected_gradients.value))

/opt/conda/lib/python3.6/site-packages/tensorflow/python/eager/backprop.py in _gradient_function(op_name, attr_tuple, num_inputs, inputs, outputs, out_grads)
129 return [None] * num_inputs
130
--&gt; 131 return grad_fn(mock_op, *out_grads)
132
133

/opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/array_grad.py in _ConcatGradV2(op, grad)
220 def _ConcatGradV2(op, grad):
221 return _ConcatGradHelper(
--&gt; 222 op, grad, start_value_index=0, end_value_index=-1, dim_index=-1)
223
224

/opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/array_grad.py in _ConcatGradHelper(op, grad, start_value_index, end_value_index, dim_index)
115 out_grads = array_ops.split(grad, sizes, non_neg_concat_dim)
116 else:
--&gt; 117 if constant_op.is_constant(concat_dim):
118 # If concat_dim is a constant defined in a different context,
119 # then we duplicate it in the current context to avoid passing it

/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py in is_constant(tensor_or_op)
293 def is_constant(tensor_or_op):
294 if isinstance(tensor_or_op, ops.Tensor):
--&gt; 295 op = tensor_or_op.op
296 else:
297 op = tensor_or_op

/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in op(self)
925 def op(self):
926 raise AttributeError(
--&gt; 927 "Tensor.op is meaningless when eager execution is enabled.")
<span class="pl-c1">928</span>
<span class="pl-c1">929</span> @<span class="pl-v">Property</span>

<span class="pl-v">AttributeError</span>: <span class="pl-v">Tensor</span>.<span class="pl-s1">op</span> <span class="pl-c1">is</span> <span class="pl-s1">meaningless</span> <span class="pl-s1">when</span> <span class="pl-s1">eager</span> <span class="pl-s1">execution</span> <span class="pl-c1">is</span> <span class="pl-s1">enabled</span>.</pre></div>
<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>: yes</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>: Linux Ubuntu 16.04</li>
<li><strong>TensorFlow installed from (source or binary)</strong>:  using nvidia containter: <a href="https://docs.nvidia.com/deeplearning/dgx/tensorflow-release-notes/rel_18.06.html#rel_18.06" rel="nofollow">https://docs.nvidia.com/deeplearning/dgx/tensorflow-release-notes/rel_18.06.html#rel_18.06</a></li>
<li><strong>TensorFlow version (use command below)</strong>: 1.8.0</li>
<li><strong>Python version</strong>: 3.5</li>
<li><strong>Bazel version (if compiling from source)</strong>: N/A</li>
<li><strong>GCC/Compiler version (if compiling from source)</strong>:N/A</li>
<li><strong>CUDA/cuDNN version</strong>: 9.0.176</li>
<li><strong>GPU model and memory</strong>: nvidia tesla v100</li>
<li><strong>Exact command to reproduce</strong>:N/A</li>
</ul>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">I'm training an network using tf.estimator where I load a set of variables from an existing checkpoint. This is done using the warm start-up option from the estimator class. While the code works when running on a single GPU, It generates an error when I use the MirroredStrategy.</p>
<h3 dir="auto">Source code / logs</h3>
<p dir="auto">The code:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ws = tf.estimator.WarmStartSettings(ckpt_to_initialize_from=&quot;init_checkpoint.ckpt&quot;),
                                                vars_to_warm_start=&quot;^((?!Logits).)*$&quot;
                                                )
self.trainer = tf.estimator.Estimator(
            model_fn=self.get_model_fn,  # First-class function
            params=None,  # HParams are passed using the config file
            config=run_config,
            warm_start_from=ws
        )"><pre class="notranslate"><code class="notranslate">ws = tf.estimator.WarmStartSettings(ckpt_to_initialize_from="init_checkpoint.ckpt"),
                                                vars_to_warm_start="^((?!Logits).)*$"
                                                )
self.trainer = tf.estimator.Estimator(
            model_fn=self.get_model_fn,  # First-class function
            params=None,  # HParams are passed using the config file
            config=run_config,
            warm_start_from=ws
        )
</code></pre></div>
<p dir="auto">This generate the following trace:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
  File &quot;train.py&quot;, line 37, in &lt;module&gt;
    experimenter.run_training_experiment(config)
  File &quot;/media/local/BDA_tf_framework/neuralnetwork/trainingexperimenter.py&quot;, line 39, in run_training_experiment
    tf.estimator.train_and_evaluate(self.trainer, self.training_specs, self.eval_specs)
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/training.py&quot;, line 439, in train_and_evaluate
    executor.run()
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/training.py&quot;, line 518, in run
    self.run_local()
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/training.py&quot;, line 650, in run_local
    hooks=train_hooks)
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py&quot;, line 363, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py&quot;, line 841, in _train_model
    return self._train_model_distributed(input_fn, hooks, saving_listeners)
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py&quot;, line 977, in _train_model_distributed
    saving_listeners)
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py&quot;, line 986, in _train_with_estimator_spec
    warm_starting_util.warm_start(*self._warm_start_settings)
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/warm_starting_util.py&quot;, line 342, in warm_start
    _warm_start_var(variable, ckpt_to_initialize_from, prev_var_name)
  File &quot;/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/warm_starting_util.py&quot;, line 139, in _warm_start_var
    &quot;PartitionedVariable, but is {}&quot;.format(type(var)))
TypeError: var MUST be one of the following: a Variable, list of Variable or PartitionedVariable, but is &lt;class 'tensorflow.contrib.distribute.python.values.MirroredVariable'&gt;
Makefile:19: recipe for target 'train_gpu' failed
make: *** [train_gpu] Error 1"><pre class="notranslate"><code class="notranslate">Traceback (most recent call last):
  File "train.py", line 37, in &lt;module&gt;
    experimenter.run_training_experiment(config)
  File "/media/local/BDA_tf_framework/neuralnetwork/trainingexperimenter.py", line 39, in run_training_experiment
    tf.estimator.train_and_evaluate(self.trainer, self.training_specs, self.eval_specs)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/training.py", line 439, in train_and_evaluate
    executor.run()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/training.py", line 518, in run
    self.run_local()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/training.py", line 650, in run_local
    hooks=train_hooks)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 363, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 841, in _train_model
    return self._train_model_distributed(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 977, in _train_model_distributed
    saving_listeners)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 986, in _train_with_estimator_spec
    warm_starting_util.warm_start(*self._warm_start_settings)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/warm_starting_util.py", line 342, in warm_start
    _warm_start_var(variable, ckpt_to_initialize_from, prev_var_name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/warm_starting_util.py", line 139, in _warm_start_var
    "PartitionedVariable, but is {}".format(type(var)))
TypeError: var MUST be one of the following: a Variable, list of Variable or PartitionedVariable, but is &lt;class 'tensorflow.contrib.distribute.python.values.MirroredVariable'&gt;
Makefile:19: recipe for target 'train_gpu' failed
make: *** [train_gpu] Error 1
</code></pre></div>