<p dir="auto">For smaller size matrix tf.reduce_mean works fine.<br>
But for larger size matrix floating point results are not accurate.</p>
<p dir="auto">I am running on CUDA 7.0, Titan X.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import tensorflow as tf

mat3 = tf.get_variable(&quot;mat3&quot;, [4500,4500], initializer=tf.constant_initializer(-1.))
reduce_mat3_ref = tf.reduce_mean(mat3)
with tf.Session('') as sess:
    tf.initialize_all_variables().run()
    reduce1 = sess.run(reduce_mat3_ref)
    print(reduce1) # on CPU this is -1.0, on GPU this produces -1.00006
    assert(reduce1 == -1.) "><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">tensorflow</span> <span class="pl-k">as</span> <span class="pl-s1">tf</span>

<span class="pl-s1">mat3</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">get_variable</span>(<span class="pl-s">"mat3"</span>, [<span class="pl-c1">4500</span>,<span class="pl-c1">4500</span>], <span class="pl-s1">initializer</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-en">constant_initializer</span>(<span class="pl-c1">-</span><span class="pl-c1">1.</span>))
<span class="pl-s1">reduce_mat3_ref</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">reduce_mean</span>(<span class="pl-s1">mat3</span>)
<span class="pl-k">with</span> <span class="pl-s1">tf</span>.<span class="pl-v">Session</span>(<span class="pl-s">''</span>) <span class="pl-k">as</span> <span class="pl-s1">sess</span>:
    <span class="pl-s1">tf</span>.<span class="pl-en">initialize_all_variables</span>().<span class="pl-en">run</span>()
    <span class="pl-s1">reduce1</span> <span class="pl-c1">=</span> <span class="pl-s1">sess</span>.<span class="pl-en">run</span>(<span class="pl-s1">reduce_mat3_ref</span>)
    <span class="pl-en">print</span>(<span class="pl-s1">reduce1</span>) <span class="pl-c"># on CPU this is -1.0, on GPU this produces -1.00006</span>
    <span class="pl-k">assert</span>(<span class="pl-s1">reduce1</span> <span class="pl-c1">==</span> <span class="pl-c1">-</span><span class="pl-c1">1.</span>) </pre></div>
<p dir="auto">I'm running Tensorflow 0.9.0 installed from wheel on Python 2.7 on a K40 with CUDA 7.0.</p>
<p dir="auto">The following test case attempts to minimize the mean of a vector through gradient descent. The script finds that the vectors are equal at all steps, but the means are not. I believe the vectors being equal at all steps is pure numerical luck, since non-deterministic loss likely means non-deterministic gradient which means non-deterministic/reproducible iterative optimization. I've observed cases where training results in different final losses where the only source of non-determinism is from reduce_mean.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np
import tensorflow as tf

n_dims = 1000
n_steps = 50

np.random.seed(2016)

vec = tf.Variable(np.random.randn(n_dims).astype(np.float32))
mean = tf.reduce_mean(vec)

optimizer = tf.train.GradientDescentOptimizer(0.01)
train_step = optimizer.minimize(mean)

def generate():
    data = []
    with tf.Session() as sess:
        sess.run(tf.initialize_all_variables())

        for _ in xrange(n_steps):
            _vec, _mean, _ = sess.run([vec, mean, train_step])
            data.append((_vec, _mean))

    return [np.array([f[i] for f in data]) for i in xrange(2)]

first_vec, first_mean = generate()
second_vec, second_mean = generate()
print 'vecs equal:', np.all(first_vec == second_vec)

print 'mean equal:', np.all(first_mean == second_mean)
print 'means not equal at idxs:', np.nonzero(first_mean != second_mean)[0]"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-k">import</span> <span class="pl-s1">tensorflow</span> <span class="pl-k">as</span> <span class="pl-s1">tf</span>

<span class="pl-s1">n_dims</span> <span class="pl-c1">=</span> <span class="pl-c1">1000</span>
<span class="pl-s1">n_steps</span> <span class="pl-c1">=</span> <span class="pl-c1">50</span>

<span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">seed</span>(<span class="pl-c1">2016</span>)

<span class="pl-s1">vec</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-v">Variable</span>(<span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">randn</span>(<span class="pl-s1">n_dims</span>).<span class="pl-en">astype</span>(<span class="pl-s1">np</span>.<span class="pl-s1">float32</span>))
<span class="pl-s1">mean</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">reduce_mean</span>(<span class="pl-s1">vec</span>)

<span class="pl-s1">optimizer</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">train</span>.<span class="pl-v">GradientDescentOptimizer</span>(<span class="pl-c1">0.01</span>)
<span class="pl-s1">train_step</span> <span class="pl-c1">=</span> <span class="pl-s1">optimizer</span>.<span class="pl-en">minimize</span>(<span class="pl-s1">mean</span>)

<span class="pl-k">def</span> <span class="pl-en">generate</span>():
    <span class="pl-s1">data</span> <span class="pl-c1">=</span> []
    <span class="pl-k">with</span> <span class="pl-s1">tf</span>.<span class="pl-v">Session</span>() <span class="pl-k">as</span> <span class="pl-s1">sess</span>:
        <span class="pl-s1">sess</span>.<span class="pl-en">run</span>(<span class="pl-s1">tf</span>.<span class="pl-en">initialize_all_variables</span>())

        <span class="pl-k">for</span> <span class="pl-s1">_</span> <span class="pl-c1">in</span> <span class="pl-en">xrange</span>(<span class="pl-s1">n_steps</span>):
            <span class="pl-s1">_vec</span>, <span class="pl-s1">_mean</span>, <span class="pl-s1">_</span> <span class="pl-c1">=</span> <span class="pl-s1">sess</span>.<span class="pl-en">run</span>([<span class="pl-s1">vec</span>, <span class="pl-s1">mean</span>, <span class="pl-s1">train_step</span>])
            <span class="pl-s1">data</span>.<span class="pl-en">append</span>((<span class="pl-s1">_vec</span>, <span class="pl-s1">_mean</span>))

    <span class="pl-k">return</span> [<span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-s1">f</span>[<span class="pl-s1">i</span>] <span class="pl-k">for</span> <span class="pl-s1">f</span> <span class="pl-c1">in</span> <span class="pl-s1">data</span>]) <span class="pl-k">for</span> <span class="pl-s1">i</span> <span class="pl-c1">in</span> <span class="pl-en">xrange</span>(<span class="pl-c1">2</span>)]

<span class="pl-s1">first_vec</span>, <span class="pl-s1">first_mean</span> <span class="pl-c1">=</span> <span class="pl-en">generate</span>()
<span class="pl-s1">second_vec</span>, <span class="pl-s1">second_mean</span> <span class="pl-c1">=</span> <span class="pl-en">generate</span>()
<span class="pl-k">print</span> <span class="pl-s">'vecs equal:'</span>, <span class="pl-s1">np</span>.<span class="pl-en">all</span>(<span class="pl-s1">first_vec</span> <span class="pl-c1">==</span> <span class="pl-s1">second_vec</span>)

<span class="pl-k">print</span> <span class="pl-s">'mean equal:'</span>, <span class="pl-s1">np</span>.<span class="pl-en">all</span>(<span class="pl-s1">first_mean</span> <span class="pl-c1">==</span> <span class="pl-s1">second_mean</span>)
<span class="pl-k">print</span> <span class="pl-s">'means not equal at idxs:'</span>, <span class="pl-s1">np</span>.<span class="pl-en">nonzero</span>(<span class="pl-s1">first_mean</span> <span class="pl-c1">!=</span> <span class="pl-s1">second_mean</span>)[<span class="pl-c1">0</span>]</pre></div>
<p dir="auto">Example output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="vecs equal: True
mean equal: False
means not equal at idxs: [ 4  5 11 18 34 38 44 49]"><pre class="notranslate"><code class="notranslate">vecs equal: True
mean equal: False
means not equal at idxs: [ 4  5 11 18 34 38 44 49]
</code></pre></div>
<p dir="auto">From looking through the code, it appears the GPU mean reduction is implemented with GPU sum reduction: <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/kernels/reduction_ops_gpu.cu.cc#L43">https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/kernels/reduction_ops_gpu.cu.cc#L43</a><br>
I've confirmed my test case still triggers when I replace "reduce_mean" with "reduce_sum".</p>
<p dir="auto">The GPU sum reduction appears to be implemented using CUDA's atomicAdd: <a href="https://bitbucket.org/eigen/eigen/src/241472d2a52142e23b0b2ba5c301c6c146298fa9/unsupported/Eigen/CXX11/src/Tensor/TensorReductionCuda.h?at=default&amp;fileviewer=file-view-default#TensorReductionCuda.h-97" rel="nofollow">https://bitbucket.org/eigen/eigen/src/241472d2a52142e23b0b2ba5c301c6c146298fa9/unsupported/Eigen/CXX11/src/Tensor/TensorReductionCuda.h?at=default&amp;fileviewer=file-view-default#TensorReductionCuda.h-97</a></p>
<p dir="auto">Atomic floating point adds on GPU are the problem. Having floating point adds to the same address in an undefined order is inherently non-deterministic due to non-associativity of floating point arithmetic.</p>
<p dir="auto">This issue could be solved (and reduction performance improved) by using some sort of reduction tree to reduce within blocks, and then launching a second kernel (or doing some manual block synchronization tricks) to reduce across blocks.</p>