<p dir="auto">I made benchmark tests for distributed setup with loopback network, profiling it and found there is excessive memory copying in the client side of RecvTensor call, which is actually one of the bottleneck.</p>
<p dir="auto">Here is the code, which mainly stolen from <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/yaroslavvb/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/yaroslavvb">@yaroslavvb</a> <a href="https://gist.github.com/yaroslavvb/1124bb02a9fd4abce3d86caf2f950cb2">here</a>,</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="  with tf.device(device1):                                                      
    params = tf.get_variable(&quot;params&quot;, shape=[params_size], dtype=dtype,        
                             initializer=tf.zeros_initializer)                  
  with tf.device(device2):                                                      
    # constant node gets placed on device1 because of simple_placer             
    #    update = tf.constant(1, shape=[params_size], dtype=dtype)              
    update = tf.get_variable(&quot;update&quot;, shape=[params_size], dtype=dtype,        
                             initializer=tf.ones_initializer())                 
    add_op = params.assign(update)"><pre class="notranslate">  <span class="pl-k">with</span> <span class="pl-s1">tf</span>.<span class="pl-en">device</span>(<span class="pl-s1">device1</span>):                                                      
    <span class="pl-s1">params</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">get_variable</span>(<span class="pl-s">"params"</span>, <span class="pl-s1">shape</span><span class="pl-c1">=</span>[<span class="pl-s1">params_size</span>], <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">dtype</span>,        
                             <span class="pl-s1">initializer</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-s1">zeros_initializer</span>)                  
  <span class="pl-k">with</span> <span class="pl-s1">tf</span>.<span class="pl-en">device</span>(<span class="pl-s1">device2</span>):                                                      
    <span class="pl-c"># constant node gets placed on device1 because of simple_placer             </span>
    <span class="pl-c">#    update = tf.constant(1, shape=[params_size], dtype=dtype)              </span>
    <span class="pl-s1">update</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">get_variable</span>(<span class="pl-s">"update"</span>, <span class="pl-s1">shape</span><span class="pl-c1">=</span>[<span class="pl-s1">params_size</span>], <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">dtype</span>,        
                             <span class="pl-s1">initializer</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-en">ones_initializer</span>())                 
    <span class="pl-s1">add_op</span> <span class="pl-c1">=</span> <span class="pl-s1">params</span>.<span class="pl-en">assign</span>(<span class="pl-s1">update</span>)</pre></div>
<p dir="auto">Here is the profiling result (google perftools) with tensor size 100MB (one fact is, the throughput will degrade with the increasing of tensor size):</p>
<ul dir="auto">
<li><a href="https://github.com/tensorflow/tensorflow/files/633183/device1-prof.pdf">device1 worker profiling report</a></li>
<li><a href="https://github.com/tensorflow/tensorflow/files/633184/device2-prof.pdf">device2 worker profiling report</a></li>
</ul>
<p dir="auto">From the result, the sending side (device2) look fine, but the receiving side (device1, the grpc client) consumes too many CPU cycles for the data transfer.</p>
<p dir="auto">By the way, I made rough stats for this <a href="https://github.com/grpc/grpc/blob/d7ff4ff40071d2b486a052183e3e9f9382afb745/src/core/lib/support/slice_buffer.c#L278">memmove</a> call. For one round of 100MB tensor assignment, there are roughly 2GB data moved (actually, including the copy inside memmove, it should be 4GB copied with a naive memmove), which is 20+ times RAM bandwidth amplification (the result is an average of 100 round run, which may not precise but the scale should be ok).</p>
<p dir="auto">NOTE: Only file GitHub issues for bugs and feature requests.  All other topics will be closed.</p>
<p dir="auto">For general support from the community, see <a href="https://stackoverflow.com/questions/tagged/tensorflow" rel="nofollow">StackOverflow</a>.<br>
To make bugs and feature requests more easy to find and organize, we close issues that are deemed<br>
out of scope for GitHub Issues and point people to StackOverflow.</p>
<p dir="auto">For bugs or installation issues, please provide the following information.<br>
The more information you provide, the more easily we will be able to offer<br>
help and advice.</p>
<h3 dir="auto">What related GitHub issues or StackOverflow threads have you found by searching the web for your problem?</h3>
<p dir="auto"><code class="notranslate">tf.contrib.learn.metric_spec.MetricSpec</code> should be changed to <code class="notranslate">MetricSpec</code> in iris_monitors.py in release 1.0.0</p>
<p dir="auto">No need to import MetricSpec in iris_monitors.py it's already imported on line 24 in commit <a href="https://github.com/tensorflow/tensorflow/blob/fa4ba830f437fdb9dc1085b4d68a3bab41a16e20/tensorflow/examples/tutorials/monitors/iris_monitors.py">fa4ba830f437fdb9dc1085b4d68a3bab41a16e20</a>:<br>
<code class="notranslate">tensorflow.contrib.learn.python.learn.metric_spec import MetricSpec</code></p>
<h3 dir="auto">Environment info</h3>
<p dir="auto">Operating System:</p>
<blockquote>
<p dir="auto">tensorflow/tensorflow/examples/tutorials/monitors$ lsb_release -a<br>
No LSB modules are available.<br>
Distributor ID:	Ubuntu<br>
Description:	Ubuntu 16.04.1 LTS<br>
Release:	16.04<br>
Codename:	xenial<br>
tensorflow/tensorflow/examples/tutorials/monitors$ uname -a<br>
Linux panchito 4.4.0-62-generic-tuxonice <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="116058758" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/83" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/83/hovercard" href="https://github.com/tensorflow/tensorflow/issues/83">#83</a>~ppa1-Ubuntu SMP Thu Feb 2 23:17:45 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux</p>
</blockquote>
<p dir="auto">Installed version of CUDA and cuDNN:<br>
(please attach the output of <code class="notranslate">ls -l /path/to/cuda/lib/libcud*</code>):</p>
<p dir="auto">If installed from binary pip package, provide:</p>
<ol dir="auto">
<li>A link to the pip package you installed:</li>
<li>The output from <code class="notranslate">python -c "import tensorflow; print(tensorflow.__version__)"</code>.</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="repositories/tensorflow/tensorflow/examples/tutorials/monitors$ python -c &quot;import tensorflow; print(tensorflow.__version__)&quot;
1.0.0"><pre class="notranslate"><code class="notranslate">repositories/tensorflow/tensorflow/examples/tutorials/monitors$ python -c "import tensorflow; print(tensorflow.__version__)"
1.0.0
</code></pre></div>
<p dir="auto">If installed from source, provide</p>
<ol dir="auto">
<li>The commit hash (<code class="notranslate">git rev-parse HEAD</code>)</li>
<li>The output of <code class="notranslate">bazel version</code></li>
</ol>
<h3 dir="auto">If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)</h3>
<p dir="auto">repositories/tensorflow/tensorflow/examples/tutorials/monitors$ python iris_monitors.py<br>
Traceback (most recent call last):<br>
File "iris_monitors.py", line 116, in <br>
tf.app.run()<br>
File "/home/chidochipotle/anaconda3/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 44, in run<br>
_sys.exit(main(_sys.argv[:1] + flags_passthrough))<br>
File "iris_monitors.py", line 42, in main<br>
tf.contrib.learn.metric_spec.MetricSpec(<br>
AttributeError: module 'tensorflow.contrib.learn' has no attribute 'metric_spec'</p>
<h3 dir="auto">What other attempted solutions have you tried?</h3>
<h3 dir="auto">Logs or other output that would be helpful</h3>
<p dir="auto">(If logs are large, please upload as attachment or provide link).</p>