<h3 dir="auto">System information</h3>
<p dir="auto">System information<br>
Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes<br>
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MacOS 10.13.4<br>
TensorFlow installed from (source or binary): Binary<br>
TensorFlow version (use command below): v1.8.0-0-g93bc2e2072 1.8.0<br>
Python version: 3.5.4<br>
Bazel version (if compiling from source): 0.11.1<br>
GCC/Compiler version (if compiling from source): 9.1.0<br>
CUDA/cuDNN version: N/A<br>
GPU model and memory: N/A<br>
Exact command to reproduce: N/A</p>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">I'm running two versions of a NN which I expect to produce identical results, but they don't.</p>
<p dir="auto">The first version contains as part of the computation:</p>
<p dir="auto"><code class="notranslate">x = [0.0*a + 1.0*b, 1.0*a + 0.0*b]</code></p>
<p dir="auto">The second contains instead:</p>
<p dir="auto"><code class="notranslate">x = [b, a]</code></p>
<p dir="auto">The two runs produce different results. (a and b are both matrices).</p>
<p dir="auto">I'm using a constant seed for the random noise input, and I've verified that multiple runs of either of the two above cases produce identical results. Just changing from one case to the other produces different results. The results are not broken in either case (i.e. both cases produce reasonable output), just different.</p>
<p dir="auto">This seems like a bug to me.</p>
<p dir="auto">Thanks</p>
<p dir="auto">EDIT:</p>
<p dir="auto">I've also found that doing:</p>
<p dir="auto">x = 1.0 * a<br>
x = 1.0 * (1.0 * a)<br>
x = tf.identity(a)</p>
<p dir="auto">All give different results.</p>
<p dir="auto">Looking at this: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="275027016" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/14675" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/14675/hovercard" href="https://github.com/tensorflow/tensorflow/issues/14675">#14675</a><br>
Apparently the problem is I've made a change to the computation graph.</p>
<p dir="auto">I still think this is a bug so I'm going to leave this here. It's very frustrating when obvious things don't work as expected.</p>
<p dir="auto">I'm running TensorFlow 1.2 with CUDA.</p>
<p dir="auto">Error case:<br>
I call <code class="notranslate">tf.set_random_seed(2017)</code> and then build a graph <code class="notranslate">g1</code> that includes trainable variables and an optimizer. I create a session, run the <code class="notranslate">tf.global_variables_initializer()</code> op, and then immediately fetch the value of a scalar variable (without running any train steps, so this is the initial value of the variable). As expected, this value stays the same if I launch this process multiple times.</p>
<p dir="auto">I then build a graph <code class="notranslate">g2</code> that is identical to <code class="notranslate">g1</code> except for a slight change. <code class="notranslate">g1</code> contained <code class="notranslate">mu = tf.reduce_sum(x) / tf.size(x)[0]</code>, and <code class="notranslate">g2</code> contains <code class="notranslate">mu = tf.reduce_mean(x)</code>. <code class="notranslate">g2</code> is seeded the same way as <code class="notranslate">g1</code> and has the same variable names and shapes as <code class="notranslate">g1</code>. The only differently named tensors are those relating to the modification mentioned above. When I fetch the initial value of the same scalar variable from <code class="notranslate">g2</code>, there is a completely different value from when fetched from <code class="notranslate">g1</code>.</p>
<p dir="auto">I've tried to isolate this into a small test case but have not been successful yet. I will continue to work on this. Apologies for bug report without test case.<br>
My current intention is to workaround this with a Numpy based initialization scheme.</p>
<p dir="auto">Questions:<br>
(1) Is this expected behavior? Ideally the variables would be initialized in the same way to help make results more reproducible. In my case, the different variable initialization makes it more difficult to test that <code class="notranslate">g1</code> and <code class="notranslate">g2</code> produce the same values. If variables were initialized the same way, would be easy to see that refactoring the mean computation in the graph did not break anything.</p>
<p dir="auto">(2) Any idea why this occurs? Perhaps relevantly, <code class="notranslate">tf.make_template</code> is used within this graph. My current (evidence-free) hunch is the small change in graph causes a variable to move from CPU resident to GPU resident and caused a different PRNG kernel (provided with the same seed) to be used.</p>