<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">The tensor.clamp function returns the upper bound when applied to a nan array on the GPU.  On the CPU it returns nan instead.</p>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import numpy as np
import torch
t = torch.FloatTensor([np.nan])
t.clamp(-10, 10)"><pre class="notranslate"><code class="notranslate">import numpy as np
import torch
t = torch.FloatTensor([np.nan])
t.clamp(-10, 10)
</code></pre></div>
<p dir="auto"><code class="notranslate">Out[26]: tensor([nan])</code></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="t.cuda().clamp(-10, 10)"><pre class="notranslate"><code class="notranslate">t.cuda().clamp(-10, 10)
</code></pre></div>
<p dir="auto"><code class="notranslate">Out[25]: tensor([10.], device='cuda:0')</code></p>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">I would expect to have the behavior be agnostic to the computational substrate.  Regardless, returning the upper bound on nan is completely arbitrary and can't possibly be the preferred solution.</p>
<h2 dir="auto">Environment</h2>
<p dir="auto">PyTorch version: 1.0.0<br>
Is debug build: No<br>
CUDA used to build PyTorch: 8.0.61</p>
<p dir="auto">OS: Ubuntu 18.10<br>
GCC version: (Ubuntu 8.2.0-7ubuntu1) 8.2.0<br>
CMake version: Could not collect</p>
<p dir="auto">Python version: 3.7<br>
Is CUDA available: Yes<br>
CUDA runtime version: Could not collect<br>
GPU models and configuration: GPU 0: GeForce GTX 1080 Ti<br>
Nvidia driver version: 415.27<br>
cuDNN version: Could not collect</p>
<p dir="auto">Versions of relevant libraries:<br>
[pip3] numpy==1.15.4<br>
[pip3] numpydoc==0.8.0<br>
[pip3] torch==1.0.0<br>
[pip3] torchvision==0.2.1<br>
[conda] blas                      1.0                         mkl<br>
[conda] cuda80                    1.0                           0    soumith<br>
[conda] cuda90                    1.0                  h6433d27_0    pytorch<br>
[conda] cuda91                    1.0                  h4c16780_0    pytorch<br>
[conda] libmklml                  2018.0.3                      0<br>
[conda] mkl                       2019.1                      144<br>
[conda] mkl-dnn                   0.14                          2    intel<br>
[conda] mkl-service               1.1.2            py37he904b0f_5<br>
[conda] mkl_fft                   1.0.6            py37hd81dba3_0<br>
[conda] mkl_random                1.0.2            py37hd81dba3_0<br>
[conda] pytorch                   1.0.0           py3.7_cuda8.0.61_cudnn7.1.2_1  [cuda80]  pytorch<br>
[conda] torchvision               0.2.1                      py_2    pytorch<br>
[conda] torchvision               0.2.1                     </p>
<p dir="auto">The behavior of clamp is not consistent on CPU and GPU when nan's are involved. See the following code snippet</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import torch
a = torch.tensor([float('nan')])
print(a.clamp(0, 1)) # prints nan
print(a.cuda().clamp(0, 1)) # prints 1"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">torch</span>
<span class="pl-s1">a</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">tensor</span>([<span class="pl-en">float</span>(<span class="pl-s">'nan'</span>)])
<span class="pl-en">print</span>(<span class="pl-s1">a</span>.<span class="pl-en">clamp</span>(<span class="pl-c1">0</span>, <span class="pl-c1">1</span>)) <span class="pl-c"># prints nan</span>
<span class="pl-en">print</span>(<span class="pl-s1">a</span>.<span class="pl-en">cuda</span>().<span class="pl-en">clamp</span>(<span class="pl-c1">0</span>, <span class="pl-c1">1</span>)) <span class="pl-c"># prints 1</span></pre></div>
<p dir="auto">This doesn't happen if we clamp only with a min or max, but only when both are passed.</p>
<p dir="auto">I believe the problem is in </p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/pytorch/pytorch/blob/517c7c98610402e2746586c78987c64c28e024aa/aten/src/THC/THCTensorMathPointwise.cuh#L309">pytorch/aten/src/THC/THCTensorMathPointwise.cuh</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 309
      in
      <a data-pjax="true" class="commit-tease-sha" href="/pytorch/pytorch/commit/517c7c98610402e2746586c78987c64c28e024aa">517c7c9</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L309" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="309"></td>
          <td id="LC309" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> T val = THCNumerics&lt;T&gt;::<span class="pl-c1">lt</span>(*in, maxValue) ? *in : maxValue; </td>
        </tr>
    </tbody></table>
  </div>
</div>
 . If we compare it to the CPU implementation in <div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/pytorch/pytorch/blob/517c7c98610402e2746586c78987c64c28e024aa/aten/src/TH/generic/THTensorMath.cpp#L122">pytorch/aten/src/TH/generic/THTensorMath.cpp</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 122
      in
      <a data-pjax="true" class="commit-tease-sha" href="/pytorch/pytorch/commit/517c7c98610402e2746586c78987c64c28e024aa">517c7c9</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L122" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="122"></td>
          <td id="LC122" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> rp[i] = (tp[i] &lt; min_value) ? min_value : (tp[i] &gt; max_value ? max_value : tp[i]); </td>
        </tr>
    </tbody></table>
  </div>
</div>
, we see that the order of the ternary operator is inversed, which doesn't propagate nans on the CUDA case.<p></p>
<p dir="auto">A simple fix would be to invert the order of the ternary operator in the CUDA kernel.</p>
<p dir="auto">Thanks <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/fmassa/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/fmassa">@fmassa</a> for helping me out identifying the root cause of the issue.</p>