<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow):Yes</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04):MS Windows10 X64 1809 build17763</li>
<li>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A</li>
<li>TensorFlow installed from (source or binary):binary(use pip)</li>
<li>TensorFlow version (use command below):v1.13.1-0-g6612da8951</li>
<li>Python version:3.6.7</li>
<li>Bazel version (if compiling from source):N/A</li>
<li>GCC/Compiler version (if compiling from source):N/A</li>
<li>CUDA/cuDNN version:10.0/7.5.1</li>
<li>GPU model and memory:NVIDIA Geforce RTX2080TI 11GB x2</li>
</ul>
<p dir="auto"><strong>Describe the current behavior</strong></p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="def _model_fn(...):
  logits, _ = _models.build_model(
          features,
          model_name=FLAGS.model_name,
          training=is_training,
          override_params=override_params)
......
  global_step = tf.train.get_global_step()
  if has_moving_average_decay:
    ema = tf.train.ExponentialMovingAverage(
        decay=FLAGS.moving_average_decay, num_updates=global_step)
    ema_vars = tf.trainable_variables() + tf.get_collection('moving_vars')
    for v in tf.global_variables():
      if 'moving_mean' in v.name or 'moving_variance' in v.name:
        ema_vars.append(v)
    ema_vars = list(set(ema_vars))
......
  if has_moving_average_decay:
      with tf.control_dependencies([train_op]):
        train_op = ema.apply(ema_vars)
......
def main():
......
  if FLAGS.num_gpus &lt;= 1:
    distribution_strategy = None
  else:
    distribution_strategy = tf.contrib.distribute.MirroredStrategy(num_gpus=FLAGS.num_gpus,                                                                   cross_device_ops=tf.contrib.distribute.AllReduceCrossDeviceOps('hierarchical_copy', num_packs=2))
  config = tf.estimator.RunConfig(
      model_dir=FLAGS.model_dir,
      train_distribute=distribution_strategy,
      save_checkpoints_steps=save_checkpoints_steps,
      log_step_count_steps=FLAGS.log_step_count_steps,
      session_config=tf.ConfigProto(
          allow_soft_placement=True,
          graph_options=tf.GraphOptions(
              rewrite_options=rewriter_config_pb2.RewriterConfig(
                  disable_meta_optimizer=True))),)
  model_est = tf.estimator.Estimator(
                        model_fn=_model_fn,
                        config=config,
                        params=params
  )
......"><pre class="notranslate"><span class="pl-k">def</span> <span class="pl-en">_model_fn</span>(...):
  <span class="pl-s1">logits</span>, <span class="pl-s1">_</span> <span class="pl-c1">=</span> <span class="pl-s1">_models</span>.<span class="pl-en">build_model</span>(
          <span class="pl-s1">features</span>,
          <span class="pl-s1">model_name</span><span class="pl-c1">=</span><span class="pl-v">FLAGS</span>.<span class="pl-s1">model_name</span>,
          <span class="pl-s1">training</span><span class="pl-c1">=</span><span class="pl-s1">is_training</span>,
          <span class="pl-s1">override_params</span><span class="pl-c1">=</span><span class="pl-s1">override_params</span>)
......
  <span class="pl-s1">global_step</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">train</span>.<span class="pl-en">get_global_step</span>()
  <span class="pl-k">if</span> <span class="pl-s1">has_moving_average_decay</span>:
    <span class="pl-s1">ema</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">train</span>.<span class="pl-v">ExponentialMovingAverage</span>(
        <span class="pl-s1">decay</span><span class="pl-c1">=</span><span class="pl-v">FLAGS</span>.<span class="pl-s1">moving_average_decay</span>, <span class="pl-s1">num_updates</span><span class="pl-c1">=</span><span class="pl-s1">global_step</span>)
    <span class="pl-s1">ema_vars</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">trainable_variables</span>() <span class="pl-c1">+</span> <span class="pl-s1">tf</span>.<span class="pl-en">get_collection</span>(<span class="pl-s">'moving_vars'</span>)
    <span class="pl-k">for</span> <span class="pl-s1">v</span> <span class="pl-c1">in</span> <span class="pl-s1">tf</span>.<span class="pl-en">global_variables</span>():
      <span class="pl-k">if</span> <span class="pl-s">'moving_mean'</span> <span class="pl-c1">in</span> <span class="pl-s1">v</span>.<span class="pl-s1">name</span> <span class="pl-c1">or</span> <span class="pl-s">'moving_variance'</span> <span class="pl-c1">in</span> <span class="pl-s1">v</span>.<span class="pl-s1">name</span>:
        <span class="pl-s1">ema_vars</span>.<span class="pl-en">append</span>(<span class="pl-s1">v</span>)
    <span class="pl-s1">ema_vars</span> <span class="pl-c1">=</span> <span class="pl-en">list</span>(<span class="pl-en">set</span>(<span class="pl-s1">ema_vars</span>))
......
  <span class="pl-k">if</span> <span class="pl-s1">has_moving_average_decay</span>:
      <span class="pl-k">with</span> <span class="pl-s1">tf</span>.<span class="pl-en">control_dependencies</span>([<span class="pl-s1">train_op</span>]):
        <span class="pl-s1">train_op</span> <span class="pl-c1">=</span> <span class="pl-s1">ema</span>.<span class="pl-en">apply</span>(<span class="pl-s1">ema_vars</span>)
......
<span class="pl-en">def</span> <span class="pl-s1">main</span>():
......
  <span class="pl-k">if</span> <span class="pl-v">FLAGS</span>.<span class="pl-s1">num_gpus</span> <span class="pl-c1">&lt;=</span> <span class="pl-c1">1</span>:
    <span class="pl-s1">distribution_strategy</span> <span class="pl-c1">=</span> <span class="pl-c1">None</span>
  <span class="pl-k">else</span>:
    <span class="pl-s1">distribution_strategy</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">contrib</span>.<span class="pl-s1">distribute</span>.<span class="pl-v">MirroredStrategy</span>(<span class="pl-s1">num_gpus</span><span class="pl-c1">=</span><span class="pl-v">FLAGS</span>.<span class="pl-s1">num_gpus</span>,                                                                   <span class="pl-s1">cross_device_ops</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-s1">contrib</span>.<span class="pl-s1">distribute</span>.<span class="pl-v">AllReduceCrossDeviceOps</span>(<span class="pl-s">'hierarchical_copy'</span>, <span class="pl-s1">num_packs</span><span class="pl-c1">=</span><span class="pl-c1">2</span>))
  <span class="pl-s1">config</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">estimator</span>.<span class="pl-v">RunConfig</span>(
      <span class="pl-s1">model_dir</span><span class="pl-c1">=</span><span class="pl-v">FLAGS</span>.<span class="pl-s1">model_dir</span>,
      <span class="pl-s1">train_distribute</span><span class="pl-c1">=</span><span class="pl-s1">distribution_strategy</span>,
      <span class="pl-s1">save_checkpoints_steps</span><span class="pl-c1">=</span><span class="pl-s1">save_checkpoints_steps</span>,
      <span class="pl-s1">log_step_count_steps</span><span class="pl-c1">=</span><span class="pl-v">FLAGS</span>.<span class="pl-s1">log_step_count_steps</span>,
      <span class="pl-s1">session_config</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-v">ConfigProto</span>(
          <span class="pl-s1">allow_soft_placement</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,
          <span class="pl-s1">graph_options</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-v">GraphOptions</span>(
              <span class="pl-s1">rewrite_options</span><span class="pl-c1">=</span><span class="pl-s1">rewriter_config_pb2</span>.<span class="pl-v">RewriterConfig</span>(
                  <span class="pl-s1">disable_meta_optimizer</span><span class="pl-c1">=</span><span class="pl-c1">True</span>))),)
  <span class="pl-s1">model_est</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">estimator</span>.<span class="pl-v">Estimator</span>(
                        <span class="pl-s1">model_fn</span><span class="pl-c1">=</span><span class="pl-s1">_model_fn</span>,
                        <span class="pl-s1">config</span><span class="pl-c1">=</span><span class="pl-s1">config</span>,
                        <span class="pl-s1">params</span><span class="pl-c1">=</span><span class="pl-s1">params</span>
  )
......</pre></div>
<p dir="auto">I have find a bug when use ExponentialMovingAverage with distribute.Strategy.These codes work fine without using a distribution strategy.But when using a distribution strategy (such as MirroredStrategy), an error is reported:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
  File &quot;C:\Users\admin\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\distribute\shared_variable_creator.py&quot;, line 90, in reuse_variable
    v = shared_variable_store[canonical_name][variable_index]
KeyError: 'mnasnet-a1/mnasnet/mnas_stem/conv2d/kernel/replica/ExponentialMovingAverage/'"><pre class="notranslate"><code class="notranslate">Traceback (most recent call last):
  File "C:\Users\admin\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\distribute\shared_variable_creator.py", line 90, in reuse_variable
    v = shared_variable_store[canonical_name][variable_index]
KeyError: 'mnasnet-a1/mnasnet/mnas_stem/conv2d/kernel/replica/ExponentialMovingAverage/'
</code></pre></div>
<p dir="auto">By debugging I can see that it works fine when ema.apply() is executed on the first GPU.But when executed on the second GPU, it will report an error.This is because MirroredVariable/ReplicaLocalVariable has been created when building the model in _model_fn().So on the second GPU, TF tries to create the variable 'conv2d/kernel/replica/ExponentialMovingAverage/'.But it doesn't exist at all, 'replica' should appear at the end of the variable name.<br>
I found a temporary solution by modifying the shared_variable_creator:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="  def reuse_variable(next_creator, *args, **kwargs):
    &quot;&quot;&quot;Re-use existing variable from store with same name (in order).&quot;&quot;&quot;
    del next_creator, args
    name = kwargs.get(&quot;name&quot;)
    canonical_name = _canonicalize_variable_name(name)
    replica_index = canonical_name.find('replica/')
    if replica_index != -1:
      canonical_name = canonical_name[0:replica_index] + canonical_name[replica_index+8:]
    try:
      variable_index = variable_scope_access_index.get(canonical_name, 0)
      v = shared_variable_store[canonical_name][variable_index]
      # TODO(priyag): Make this variable re-use more robust by adding checks
      # that the requested shape and dtype match the existing variable.
      variable_scope_access_index[canonical_name] = variable_index + 1
      return v
    except (KeyError, IndexError):
      raise RuntimeError(
          &quot;Tried to create variable {} with mismatching name on device {}&quot;.
          format(name, device_id))

  if device_id == 0:
    return create_new_variable
  else:
    return reuse_variable"><pre class="notranslate">  <span class="pl-k">def</span> <span class="pl-en">reuse_variable</span>(<span class="pl-s1">next_creator</span>, <span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>):
    <span class="pl-s">"""Re-use existing variable from store with same name (in order)."""</span>
    <span class="pl-k">del</span> <span class="pl-s1">next_creator</span>, <span class="pl-s1">args</span>
    <span class="pl-s1">name</span> <span class="pl-c1">=</span> <span class="pl-s1">kwargs</span>.<span class="pl-en">get</span>(<span class="pl-s">"name"</span>)
    <span class="pl-s1">canonical_name</span> <span class="pl-c1">=</span> <span class="pl-en">_canonicalize_variable_name</span>(<span class="pl-s1">name</span>)
    <span class="pl-s1">replica_index</span> <span class="pl-c1">=</span> <span class="pl-s1">canonical_name</span>.<span class="pl-en">find</span>(<span class="pl-s">'replica/'</span>)
    <span class="pl-k">if</span> <span class="pl-s1">replica_index</span> <span class="pl-c1">!=</span> <span class="pl-c1">-</span><span class="pl-c1">1</span>:
      <span class="pl-s1">canonical_name</span> <span class="pl-c1">=</span> <span class="pl-s1">canonical_name</span>[<span class="pl-c1">0</span>:<span class="pl-s1">replica_index</span>] <span class="pl-c1">+</span> <span class="pl-s1">canonical_name</span>[<span class="pl-s1">replica_index</span><span class="pl-c1">+</span><span class="pl-c1">8</span>:]
    <span class="pl-k">try</span>:
      <span class="pl-s1">variable_index</span> <span class="pl-c1">=</span> <span class="pl-s1">variable_scope_access_index</span>.<span class="pl-en">get</span>(<span class="pl-s1">canonical_name</span>, <span class="pl-c1">0</span>)
      <span class="pl-s1">v</span> <span class="pl-c1">=</span> <span class="pl-s1">shared_variable_store</span>[<span class="pl-s1">canonical_name</span>][<span class="pl-s1">variable_index</span>]
      <span class="pl-c"># TODO(priyag): Make this variable re-use more robust by adding checks</span>
      <span class="pl-c"># that the requested shape and dtype match the existing variable.</span>
      <span class="pl-s1">variable_scope_access_index</span>[<span class="pl-s1">canonical_name</span>] <span class="pl-c1">=</span> <span class="pl-s1">variable_index</span> <span class="pl-c1">+</span> <span class="pl-c1">1</span>
      <span class="pl-k">return</span> <span class="pl-s1">v</span>
    <span class="pl-k">except</span> (<span class="pl-v">KeyError</span>, <span class="pl-v">IndexError</span>):
      <span class="pl-k">raise</span> <span class="pl-v">RuntimeError</span>(
          <span class="pl-s">"Tried to create variable {} with mismatching name on device {}"</span>.
          <span class="pl-en">format</span>(<span class="pl-s1">name</span>, <span class="pl-s1">device_id</span>))

  <span class="pl-k">if</span> <span class="pl-s1">device_id</span> <span class="pl-c1">==</span> <span class="pl-c1">0</span>:
    <span class="pl-k">return</span> <span class="pl-s1">create_new_variable</span>
  <span class="pl-k">else</span>:
    <span class="pl-k">return</span> <span class="pl-s1">reuse_variable</span></pre></div>
<p dir="auto">But this looks more like a patch than a solution.I hope someone can review this bug and fix it.<br>
<strong>Describe the expected behavior</strong></p>
<p dir="auto"><strong>Code to reproduce the issue</strong><br>
Provide a reproducible test case that is the bare minimum necessary to generate the problem.</p>
<p dir="auto"><strong>Other info / logs</strong><br>
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.</p>
<p dir="auto">I have Ubuntu 14.04 system I installed tensor flow using<br>
pip install <a href="https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.5.0-cp27-none-linux_x86_64.whl" rel="nofollow">https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.5.0-cp27-none-linux_x86_64.whl</a></p>
<p dir="auto">then when I try to import tensorflow in my python window, I get the error:<br>
<strong>init</strong>() got an unexpected keyword argument 'syntax'</p>
<p dir="auto">Is it a protobuf version problem ?<br>
I have $ protoc --version<br>
libprotoc 2.5.0</p>