<p dir="auto">I've been trying to normalize some vectors by dividing by their sum, but have noticed that this doesn't always result in vectors that sum to one.</p>
<p dir="auto">For example:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="foo = np.random.rand(100)*100 #random floats between 0-100
print(sess.run(tf.reduce_sum(foo/tf.reduce_sum(foo)))) #should return 1
print((foo/foo.sum()).sum()) #does return 1"><pre class="notranslate"><code class="notranslate">foo = np.random.rand(100)*100 #random floats between 0-100
print(sess.run(tf.reduce_sum(foo/tf.reduce_sum(foo)))) #should return 1
print((foo/foo.sum()).sum()) #does return 1
</code></pre></div>
<p dir="auto">While I realize that this is possible with finite precision floating point operations, I find it odd how readily it occurs, especially when the same operation in numpy doesn't encounter this issue. Is this a known issue? I've also noticed that tf.nn.softmax doesn't appear to suffer from this problem, is there a different op I need to use to ensure proper normalization?</p>
<p dir="auto">It would be nice to be able to use numerically stable summation methods like</p>
<ul dir="auto">
<li>pairwise summation</li>
<li>Kahan summation</li>
</ul>
<p dir="auto">for applications where numerical accuracy is important.<br>
This could work with an optional <code class="notranslate">stable</code> keyword argument to <code class="notranslate">tf.reduce_sum</code> or via a <code class="notranslate">tf.stable_reduce_sum</code> op.</p>
<p dir="auto">I think implementing Kahan summation in the Eigen tensor library wouldn't be difficult at all, we would simply have to add a stateful <code class="notranslate">KahanSumReducer</code>.<br>
It should also be possible to provide a vectorized version.<br>
Pairwise summation might be more difficult, as the reduction code would have to be touched.</p>