<p dir="auto">Hi,</p>
<p dir="auto">I've just noticed a dangerous "silent overflow" in Numpy when used in Jupyter notebooks. I understand there were other discussions about similar silent overflows, but this has really put me on the wrong foot, and I believe there must be millions of calculations out there silently failing, and thus giving wrong results.</p>
<p dir="auto">Here is a function that stretches the values of an array between a given minimum and maximum:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import numpy as np
from scipy import ndimage

def stretcher(a, new_min, new_max):
    old_min = ndimage.minimum(a)
    old_max = ndimage.maximum(a)
    if old_min == old_max:
        return a - old_min + new_min
    return new_min + (a - old_min) * (new_max - new_min) / (old_max - old_min)"><pre class="notranslate"><code class="notranslate">import numpy as np
from scipy import ndimage

def stretcher(a, new_min, new_max):
    old_min = ndimage.minimum(a)
    old_max = ndimage.maximum(a)
    if old_min == old_max:
        return a - old_min + new_min
    return new_min + (a - old_min) * (new_max - new_min) / (old_max - old_min)
</code></pre></div>
<p dir="auto">If we execute this code that stretches the values to a positive interval:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="a = np.array([0, -182, -10, 0], np.int16)
b = stretcher(a, new_min=np.int16(0), new_max=np.int16(182))
print(b)"><pre class="notranslate"><code class="notranslate">a = np.array([0, -182, -10, 0], np.int16)
b = stretcher(a, new_min=np.int16(0), new_max=np.int16(182))
print(b)
</code></pre></div>
<p dir="auto">we get this result:</p>
<p dir="auto"><code class="notranslate">[-178.08791209    0.          172.         -178.08791209]</code></p>
<p dir="auto">A very logic result! But not for earthlings like me... You see, I understand that 0 + 182 * 182 / 182 returns -178.08791209 because it is interpreted as 0 + (182 * 182) / 182 , and 182 * 182 = 33124, which is greater than the maximum that np.int16 can hold (32767). However, I would prefer the execution to fail, so that I know I must take extra measures in my code to catch such overflows.</p>
<p dir="auto">Moreover, <code class="notranslate">print(b.dtype)</code> returns <code class="notranslate">float64</code>, so one would never think that 182*182 did not fit in a 64 bit float...</p>
<p dir="auto">The correct result is returned by this slightly modified statement (notice the grouping of the division operation):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="return new_min + (a - old_min) * ((new_max - new_min) / (old_max - old_min))
[182.   0. 172. 182.]"><pre class="notranslate"><code class="notranslate">return new_min + (a - old_min) * ((new_max - new_min) / (old_max - old_min))
[182.   0. 172. 182.]
</code></pre></div>
<p dir="auto">The result type is <code class="notranslate">float32</code> in this case (don't ask me why).</p>
<p dir="auto">The fact that <code class="notranslate">x*y/z</code> returns a different result than<code class="notranslate"> x*(y/z)</code> when z is not zero (nor close to zero) and x, y, and z are quite moderate values is also a concerning issue.</p>
<p dir="auto">I don't know how Numpy optimizes calculations, but when you see a division (which seems to automatically convert the result to float), you might want to convert all values from that statement to float before any operation takes place.</p>
<p dir="auto">I see that if I run this:</p>
<p dir="auto"><code class="notranslate">print(np.int16(182)*np.int16(182))</code></p>
<p dir="auto">in a Jupyter notebook I do get the result -32412 and a warning:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[path to Python here]\lib\site-packages\ipykernel\__main__.py:1: RuntimeWarning: overflow encountered in short_scalars
  if __name__ == '__main__':"><pre class="notranslate"><code class="notranslate">[path to Python here]\lib\site-packages\ipykernel\__main__.py:1: RuntimeWarning: overflow encountered in short_scalars
  if __name__ == '__main__':
</code></pre></div>
<p dir="auto">Currently there is no warning in Jupyter notebooks when the same operation is inside a function. It is so easy to believe that everything works fine!</p>
<p dir="auto">You may say that it's Jupyter's fault, but if the integer overflow would have caused an error, I would have seen the problem immediately in Jupyter, too.</p>
<p dir="auto">You may also say that I should not work with int16, but with int32 or int64. However, I am constrained by the libraries I work with. Some of them accept only uint8, others int16, and others int32. I already do a lot of conversions back and forth. If I have to convert each time I fear an overflow may happen the code will be messy and slow. I currently cast just before feeding the data to those libraries that accept a single type of data.</p>
<p dir="auto">By the way, wasn't Python supposed to automatically cast to wider range types when needed? I understand that Numpy tries to keep the type of the result unchanged if the two operands have the same type, but in this case the type will change anyway, so why imposing a constraint (resulting in an error) when the constraint is lifted automatically one step further in the same statement?</p>
<p dir="auto">I think it is more logical, useful, pythonic and correct (numerically speaking) to automatically cast to higher types when needed and either issuing a warning about such cast, or returning an extra flag indicating that the type has changed, than to return a wrong result and issue a warning about it (which disappears anyway as shown above).</p>
<p dir="auto">Thanks,</p>
<p dir="auto">Andrei</p>
<p dir="auto">Simple demo:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="In [1]: np.uint8(0) - np.uint8(1)
RuntimeWarning: overflow encountered in ubyte_scalars
Out[1]: 255

In [2]: np.uint8(0)[...] - np.uint8(1)[...]
Out[2]: 255"><pre class="notranslate"><span class="pl-v">In</span> [<span class="pl-c1">1</span>]: <span class="pl-s1">np</span>.<span class="pl-en">uint8</span>(<span class="pl-c1">0</span>) <span class="pl-c1">-</span> <span class="pl-s1">np</span>.<span class="pl-en">uint8</span>(<span class="pl-c1">1</span>)
<span class="pl-v">RuntimeWarning</span>: <span class="pl-s1">overflow</span> <span class="pl-s1">encountered</span> <span class="pl-c1">in</span> <span class="pl-s1">ubyte_scalars</span>
<span class="pl-v">Out</span>[<span class="pl-c1">1</span>]: <span class="pl-c1">255</span>

<span class="pl-v">In</span> [<span class="pl-c1">2</span>]: <span class="pl-s1">np</span>.<span class="pl-en">uint8</span>(<span class="pl-c1">0</span>)[...] <span class="pl-c1">-</span> <span class="pl-s1">np</span>.<span class="pl-en">uint8</span>(<span class="pl-c1">1</span>)[...]
<span class="pl-v">Out</span>[<span class="pl-c1">2</span>]: <span class="pl-c1">255</span></pre></div>
<p dir="auto">Should we even warn at all for unsigned integers?</p>