<h3 dir="auto">System information</h3>
<p dir="auto">OS X with a CPU libtensorflow.so built off master (<a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/tensorflow/tensorflow/commit/2336cdf7f00434c24f1eb394f255d7de071bdc08/hovercard" href="https://github.com/tensorflow/tensorflow/commit/2336cdf7f00434c24f1eb394f255d7de071bdc08"><tt>2336cdf</tt></a>)</p>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto"><code class="notranslate">TF_AddGradients</code> reports</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="No gradient defined for op: Add. Please see https://www.tensorflow.org/code/tensorflow/cc/gradients/README.md for instructions on how to add C++ gradients."><pre class="notranslate"><code class="notranslate">No gradient defined for op: Add. Please see https://www.tensorflow.org/code/tensorflow/cc/gradients/README.md for instructions on how to add C++ gradients.
</code></pre></div>
<p dir="auto">when asked to compute dz/dx for z=x+y (see code example).</p>
<p dir="auto">and yet <code class="notranslate">Add</code> has a <a href="https://github.com/tensorflow/tensorflow/blob/2336cdf7f00434c24f1eb394f255d7de071bdc08/tensorflow/core/ops/math_grad.cc#L372">registered gradient</a>. <code class="notranslate">Mul</code> also doesn't work, although gradients of other operations (like <code class="notranslate">Sin</code>) do work.</p>
<h3 dir="auto">Source code / logs</h3>
<div class="highlight highlight-source-c notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="#include &quot;c_api.h&quot;

void check_status(TF_Status* status) {
  auto code = TF_GetCode(status);
  if (code != TF_OK) {
    cout&lt;&lt;TF_Message(status)&lt;&lt;endl;
    exit(0);
  }
}

int main() {
  auto graph = TF_NewGraph();
  auto x_desc = TF_NewOperation(graph, &quot;Placeholder&quot;, &quot;x&quot;);
  TF_SetAttrType(x_desc, &quot;dtype&quot;, TF_FLOAT);
  auto status = TF_NewStatus();
  auto x = TF_FinishOperation(x_desc, status);
  auto y_desc = TF_NewOperation(graph, &quot;Placeholder&quot;, &quot;y&quot;);
  TF_SetAttrType(y_desc, &quot;dtype&quot;, TF_FLOAT);
  auto y = TF_FinishOperation(y_desc, status);
  auto z_desc = TF_NewOperation(graph, &quot;Add&quot;, &quot;z&quot;);
  TF_AddInput(z_desc, {x, 0});
  TF_AddInput(z_desc, {y, 0});
  auto z = TF_FinishOperation(z_desc, status);
  TF_Output output = {z, 0};
  TF_Output input = {x, 0};
  TF_Output grad;
  TF_AddGradients(graph, &amp;output, 1, &amp;input, 1, NULL, status, &amp;grad);
  check_status(status);
  return 0;
}
"><pre class="notranslate"><span class="pl-k">#include</span> <span class="pl-s">"c_api.h"</span>

<span class="pl-smi">void</span> <span class="pl-en">check_status</span>(<span class="pl-smi">TF_Status</span><span class="pl-c1">*</span> <span class="pl-s1">status</span>) {
  auto <span class="pl-smi">code</span> <span class="pl-c1">=</span> <span class="pl-en">TF_GetCode</span>(<span class="pl-smi">status</span>);
  <span class="pl-k">if</span> (<span class="pl-s1">code</span> <span class="pl-c1">!=</span> <span class="pl-c1">TF_OK</span>) {
    <span class="pl-s1">cout</span>&lt;&lt;<span class="pl-en">TF_Message</span>(<span class="pl-s1">status</span>)&lt;&lt;<span class="pl-s1">endl</span>;
    <span class="pl-en">exit</span>(<span class="pl-c1">0</span>);
  }
}

<span class="pl-smi">int</span> <span class="pl-en">main</span>() {
  auto <span class="pl-smi">graph</span> <span class="pl-c1">=</span> <span class="pl-en">TF_NewGraph</span>();
  auto <span class="pl-smi">x_desc</span><span class="pl-s1"></span> <span class="pl-c1">=</span> <span class="pl-en">TF_NewOperation</span>(<span class="pl-s1">graph</span>, <span class="pl-s">"Placeholder"</span>, <span class="pl-s">"x"</span>);
  <span class="pl-en">TF_SetAttrType</span>(<span class="pl-s1">x_desc</span>, <span class="pl-s">"dtype"</span>, <span class="pl-c1">TF_FLOAT</span>);
  auto <span class="pl-smi">status</span> <span class="pl-c1">=</span> <span class="pl-en">TF_NewStatus</span>();
  auto <span class="pl-smi">x</span> <span class="pl-c1">=</span> <span class="pl-en">TF_FinishOperation</span>(<span class="pl-smi">x_desc</span>, <span class="pl-smi">status</span>);
  auto <span class="pl-smi">y_desc</span><span class="pl-s1"></span> <span class="pl-c1">=</span> <span class="pl-en">TF_NewOperation</span>(<span class="pl-s1">graph</span>, <span class="pl-s">"Placeholder"</span>, <span class="pl-s">"y"</span>);
  <span class="pl-en">TF_SetAttrType</span>(<span class="pl-s1">y_desc</span>, <span class="pl-s">"dtype"</span>, <span class="pl-c1">TF_FLOAT</span>);
  auto <span class="pl-smi">y</span> <span class="pl-c1">=</span> <span class="pl-en">TF_FinishOperation</span>(<span class="pl-smi">y_desc</span>, <span class="pl-smi">status</span>);
  auto <span class="pl-smi">z_desc</span><span class="pl-s1"></span> <span class="pl-c1">=</span> <span class="pl-en">TF_NewOperation</span>(<span class="pl-s1">graph</span>, <span class="pl-s">"Add"</span>, <span class="pl-s">"z"</span>);
  <span class="pl-en">TF_AddInput</span>(<span class="pl-s1">z_desc</span>, {<span class="pl-s1">x</span>, <span class="pl-c1">0</span>});
  <span class="pl-en">TF_AddInput</span>(<span class="pl-s1">z_desc</span>, {<span class="pl-s1">y</span>, <span class="pl-c1">0</span>});
  auto <span class="pl-smi">z</span> <span class="pl-c1">=</span> <span class="pl-en">TF_FinishOperation</span>(<span class="pl-smi">z_desc</span>, <span class="pl-smi">status</span>);
  <span class="pl-smi">TF_Output</span> <span class="pl-s1">output</span> <span class="pl-c1">=</span> {<span class="pl-s1">z</span>, <span class="pl-c1">0</span>};
  <span class="pl-smi">TF_Output</span> <span class="pl-s1">input</span> <span class="pl-c1">=</span> {<span class="pl-s1">x</span>, <span class="pl-c1">0</span>};
  <span class="pl-smi">TF_Output</span> <span class="pl-s1">grad</span>;
  <span class="pl-en">TF_AddGradients</span>(<span class="pl-s1">graph</span>, <span class="pl-c1">&amp;</span><span class="pl-s1">output</span>, <span class="pl-c1">1</span>, <span class="pl-c1">&amp;</span><span class="pl-s1">input</span>, <span class="pl-c1">1</span>, <span class="pl-c1">NULL</span>, <span class="pl-s1">status</span>, <span class="pl-c1">&amp;</span><span class="pl-s1">grad</span>);
  <span class="pl-en">check_status</span>(<span class="pl-s1">status</span>);
  <span class="pl-k">return</span> <span class="pl-c1">0</span>;
}</pre></div>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">In my model, I use the FTRL Optimizer like below:</p>
<p dir="auto"><code class="notranslate">self.optimizer = tf.train.FtrlOptimizer(0.005, learning_rate_power=-0.5, initial_accumulator_value=0.1, l1_regularization_strength=1.0, l2_regularization_strength=0.00001) </code></p>
<p dir="auto">Inside the FtrlOptimizer it will create several const variables for the parameters, such as learning rate, learning rate power, etc.</p>
<p dir="auto">When I run the distributed tensorflow job, from the timeline I can see that for each session run I can see that the worker will send the above const variables to all the ps nodes. This is a cost since the variables are const and not needed to sent to ps nodes repeatedly.</p>
<p dir="auto">I was wondering is there a way to pin those const variables to the ps and save the transferring cost during each session run.</p>
<p dir="auto">Thanks.</p>