<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">While looking at issue <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="499503440" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/26971" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/26971/hovercard" href="https://github.com/pytorch/pytorch/pull/26971">#26971</a>, I wondered if torch's uniform number generators was generating too many zeros since it generates in average more than 5 zeros per 10^8 generations according to the following:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="sum((torch.rand(100000000) &lt;= 0.).sum().float() for i in range(20)) / 20"><pre class="notranslate"><span class="pl-en">sum</span>((<span class="pl-s1">torch</span>.<span class="pl-en">rand</span>(<span class="pl-c1">100000000</span>) <span class="pl-c1">&lt;=</span> <span class="pl-c1">0.</span>).<span class="pl-en">sum</span>().<span class="pl-en">float</span>() <span class="pl-k">for</span> <span class="pl-s1">i</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">20</span>)) <span class="pl-c1">/</span> <span class="pl-c1">20</span></pre></div>
<p dir="auto">On GPU, it generates in average around 3 zeros per 10^8 generators:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="sum((torch.rand(100000000, device=torch.device('cuda')) &lt;= 0.).sum().float() for i in range(20)) / 20"><pre class="notranslate"><span class="pl-en">sum</span>((<span class="pl-s1">torch</span>.<span class="pl-en">rand</span>(<span class="pl-c1">100000000</span>, <span class="pl-s1">device</span><span class="pl-c1">=</span><span class="pl-s1">torch</span>.<span class="pl-en">device</span>(<span class="pl-s">'cuda'</span>)) <span class="pl-c1">&lt;=</span> <span class="pl-c1">0.</span>).<span class="pl-en">sum</span>().<span class="pl-en">float</span>() <span class="pl-k">for</span> <span class="pl-s1">i</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">20</span>)) <span class="pl-c1">/</span> <span class="pl-c1">20</span></pre></div>
<p dir="auto">If you do the same test with NumPy, you will see that it never generates zeros:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="sum((np.random.rand(100000000).astype('float32') &lt;= 0.).sum() for i in range(20)) / 20"><pre class="notranslate"><span class="pl-en">sum</span>((<span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">rand</span>(<span class="pl-c1">100000000</span>).<span class="pl-en">astype</span>(<span class="pl-s">'float32'</span>) <span class="pl-c1">&lt;=</span> <span class="pl-c1">0.</span>).<span class="pl-en">sum</span>() <span class="pl-k">for</span> <span class="pl-s1">i</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">20</span>)) <span class="pl-c1">/</span> <span class="pl-c1">20</span></pre></div>
<p dir="auto">(if you try with <code class="notranslate">'float16'</code> instead of <code class="notranslate">'float32'</code>, you will see that it generates some zeros)<br>
Thus, I think that PyTorch's generator might be biased towards zero and this may hide other biases that I'm not aware.</p>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">See above.</p>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">In my opinion, the behavior should be similar to NumPy's unless NumPy has a bug instead.</p>
<h2 dir="auto">Environment</h2>
<p dir="auto">Collecting environment information...<br>
PyTorch version: 1.2.0<br>
Is debug build: No<br>
CUDA used to build PyTorch: 10.0.130</p>
<p dir="auto">OS: Ubuntu 18.04.3 LTS<br>
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0<br>
CMake version: Could not collect</p>
<p dir="auto">Python version: 3.6<br>
Is CUDA available: Yes<br>
CUDA runtime version: 9.1.85<br>
GPU models and configuration:<br>
GPU 0: GeForce GTX 1080 Ti<br>
GPU 1: GeForce GTX 1080 Ti<br>
GPU 2: GeForce GT 1030</p>
<p dir="auto">Nvidia driver version: 430.26<br>
cuDNN version: Could not collect</p>
<p dir="auto">Versions of relevant libraries:<br>
[pip3] msgpack-numpy==0.4.3.1<br>
[pip3] numpy==1.16.2<br>
[pip3] numpydoc==0.8.0<br>
[pip3] torch==1.2.0<br>
[pip3] torchvision==0.4.0a0+6b959ee<br>
[conda] blas                      1.0                         mkl<br>
[conda] mkl                       2019.1                      144<br>
[conda] mkl-service               1.1.2            py36he904b0f_5<br>
[conda] mkl_fft                   1.0.10           py36ha843d7b_0<br>
[conda] mkl_random                1.0.2            py36hd81dba3_0<br>
[conda] pytorch                   1.2.0           py3.6_cuda10.0.130_cudnn7.6.2_0    pytorch<br>
[conda] torch                     1.1.0                    pypi_0    pypi<br>
[conda] torchvision               0.2.1                    pypi_0    pypi</p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">When using uniformly-distributed random number generation, especially using <code class="notranslate">float32</code>, over a range including 0, the sampling code ignores the extra precision of floating-point around 0, leading to very bad sampling.</p>
<p dir="auto">The generated floats also have at most 24 (<code class="notranslate">float32</code>) or 53 (<code class="notranslate">float64</code>) bits of entropy.</p>
<p dir="auto"><strong>Consequence 1:</strong> Multiplying any PyTorch float32 <code class="notranslate">uniform_(0,1)</code> by a large integer like <code class="notranslate">2**33</code> is guaranteed to give an integer.<br>
<strong>Consequence 2:</strong> It is actually possible to generate an exact <code class="notranslate">0.0</code> using the <code class="notranslate">float32</code> <strong><code class="notranslate">uniform_(0,1)</code>,</strong> with probability 2**-24, whereas that should actually have been 2**-149.<br>
<strong>Consequence 3:</strong> It is not possible to generate a <code class="notranslate">float32</code> strictly between <code class="notranslate">0.0</code> and <code class="notranslate">2**-24</code> with <code class="notranslate">uniform_(0,1)</code>, whereas it should have been possible with probability 2**-24.</p>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto"><em>Test 1:</em> If the following is <code class="notranslate">!= 0</code>, the RNG sucks: <code class="notranslate">torch.FloatTensor(int(1e8)).uniform_().eq(0).sum()</code><br>
<em>Test 2:</em> If the following is <code class="notranslate">== -inf</code>, the RNG sucks: <code class="notranslate">torch.FloatTensor(int(1e8)).uniform_().log().sum()</code></p>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto"><em>Test 1:</em> A return of <code class="notranslate">0</code>.<br>
<em>Test 2:</em> A finite, negative value.</p>
<h2 dir="auto">Environment</h2>
<p dir="auto"><em>Not relevant.</em></p>
<ul dir="auto">
<li>PyTorch Version 1.1.0a0+c865d46, but the same problem dates back to the beginning of the history of this code, which is over a year.</li>
</ul>
<h2 dir="auto">Additional context</h2>
<p dir="auto">This has serious knock-on effects on many other distributions that use <code class="notranslate">uniform_()</code> as a basis. In particular, <code class="notranslate">log(uniform_())</code> has a 2**-24 chance of producing <code class="notranslate">-inf</code>, and <code class="notranslate">log2(uniform_())</code> cannot produce a number <code class="notranslate">&lt; -24</code> (except <code class="notranslate">-inf</code>).</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/pbelevich/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/pbelevich">@pbelevich</a></p>