<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">When I place a <code class="notranslate">torch.arange()</code> tensor on a GPU I get the following error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="RuntimeError: parallel_for failed: cudaErrorNoKernelImageForDevice: no kernel image is available for execution on the device"><pre class="notranslate"><code class="notranslate">RuntimeError: parallel_for failed: cudaErrorNoKernelImageForDevice: no kernel image is available for execution on the device
</code></pre></div>
<p dir="auto">This occurs on a system where I have a Tesla K80 and two Tesla K40c cards (also a GeForce 710 which is just for display purposes). It seems to be only related to the K40c cards and not the K80, but occurs regardless of what GPU I try to place the tensor on, if either of the K40c cards are visible via CUDA_VISIBLE_DEVICES.</p>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<p dir="auto">In a fresh conda environment, install the latest stable version of PyTorch (1.3.1) then a minimal working example is:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import torch
torch.arange(0, 10, dtype=torch.int32, device=torch.device(&quot;cuda&quot;))"><pre class="notranslate"><code class="notranslate">import torch
torch.arange(0, 10, dtype=torch.int32, device=torch.device("cuda"))
</code></pre></div>
<p dir="auto">To be clear which GPUs are causing the problem, I first set <code class="notranslate">CUDA_DEVICE_ORDER=PCI_BUS_ID</code> so that I can be sure which GPUs are visible. If I set only either of the K80 dies to be visible, then this error doesn't happen, but setting either K40c visible (and also the GeForce 710, of course) does cause this error to happen. I suspect this is related to the lower compute capability of the K40c devices.</p>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">The documentation for v1.3.1 states <a href="https://pytorch.org/docs/stable/torch.html?highlight=compute%20capability" rel="nofollow">here</a> that CUDA devices with compute capability &gt;= 3.0 should be supported. I can't find anywhere else where there is a specification of which compute capabilities are supported from the pre-compiled binaries installed through conda, so I wouldn't expect this to be an intended deprecation, unless I missed it somewhere.</p>
<h2 dir="auto">Environment</h2>
<p dir="auto">PyTorch version: 1.3.1<br>
Is debug build: No<br>
CUDA used to build PyTorch: 10.1.243</p>
<p dir="auto">OS: Manjaro Linux<br>
GCC version: (GCC) 9.2.0<br>
CMake version: Could not collect</p>
<p dir="auto">Python version: 3.6<br>
Is CUDA available: Yes<br>
CUDA runtime version: Could not collect<br>
GPU models and configuration:<br>
GPU 0: GeForce GT 710<br>
GPU 1: Tesla K80<br>
GPU 2: Tesla K80<br>
GPU 3: Tesla K40c<br>
GPU 4: Tesla K40c</p>
<p dir="auto">Nvidia driver version: 418.113<br>
cuDNN version: Could not collect</p>
<p dir="auto">Versions of relevant libraries:<br>
[pip] numpy==1.17.4<br>
[pip] torch==1.3.1<br>
[pip] torchvision==0.4.2<br>
[conda] blas                      1.0                         mkl<br>
[conda] mkl                       2019.4                      243<br>
[conda] mkl-service               2.3.0            py36he904b0f_0<br>
[conda] mkl_fft                   1.0.15           py36ha843d7b_0<br>
[conda] mkl_random                1.1.0            py36hd6b4f25_0<br>
[conda] pytorch                   1.3.1           py3.6_cuda10.1.243_cudnn7.6.3_0    pytorch<br>
[conda] torchvision               0.4.2                py36_cu101    pytorch</p>
<p dir="auto">Edit: Forgot to add that this does <em>not</em> occur if I install PyTorch 1.3.0 instead.</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ezyang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ezyang">@ezyang</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gchanan/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gchanan">@gchanan</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zou3519/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zou3519">@zou3519</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ngimel/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ngimel">@ngimel</a></p>
<h2 dir="auto">🐛 Bug</h2>
<p dir="auto">I am using a Tesla K40m, installed pytorch 1.3 with conda, using CUDA 10.1</p>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<ol dir="auto">
<li>Have a box with a Tesla K40m</li>
<li><code class="notranslate">conda install pytorch cudatoolkit -c pytorch</code></li>
<li>show cuda is available</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="python -c 'import torch; print(torch.cuda.is_available());'
&gt;&gt;&gt; True"><pre class="notranslate"><code class="notranslate">python -c 'import torch; print(torch.cuda.is_available());'
&gt;&gt;&gt; True
</code></pre></div>
<ol dir="auto">
<li>Instantiate a model and call <code class="notranslate">.forward()</code></li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
  File &quot;./baselines/get_results.py&quot;, line 395, in &lt;module&gt;
    main(args)
  File &quot;./baselines/get_results.py&quot;, line 325, in main
    log_info = eval_main(eval_args)
  File &quot;/mnt/cdtds_cluster_home/s0816700/git/midi_degradation_toolkit/baselines/eval_task.py&quot;, line 165, in main
    log_info = trainer.test(0, evaluate=True)
  File &quot;/mnt/cdtds_cluster_home/s0816700/git/midi_degradation_toolkit/mdtk/pytorch_trainers.py&quot;, line 110, in test
    evaluate=evaluate)
  File &quot;/mnt/cdtds_cluster_home/s0816700/git/midi_degradation_toolkit/mdtk/pytorch_trainers.py&quot;, line 220, in iteration
    model_output = self.model.forward(input_data, input_lengths)
  File &quot;/mnt/cdtds_cluster_home/s0816700/git/midi_degradation_toolkit/mdtk/pytorch_models.py&quot;, line 49, in forward
    self.hidden = self.init_hidden(batch_size, device=device)
  File &quot;/mnt/cdtds_cluster_home/s0816700/git/midi_degradation_toolkit/mdtk/pytorch_models.py&quot;, line 40, in init_hidden
    return (torch.randn(1, batch_size, self.hidden_dim, device=device),
RuntimeError: CUDA error: no kernel image is available for execution on the device"><pre class="notranslate"><code class="notranslate">Traceback (most recent call last):
  File "./baselines/get_results.py", line 395, in &lt;module&gt;
    main(args)
  File "./baselines/get_results.py", line 325, in main
    log_info = eval_main(eval_args)
  File "/mnt/cdtds_cluster_home/s0816700/git/midi_degradation_toolkit/baselines/eval_task.py", line 165, in main
    log_info = trainer.test(0, evaluate=True)
  File "/mnt/cdtds_cluster_home/s0816700/git/midi_degradation_toolkit/mdtk/pytorch_trainers.py", line 110, in test
    evaluate=evaluate)
  File "/mnt/cdtds_cluster_home/s0816700/git/midi_degradation_toolkit/mdtk/pytorch_trainers.py", line 220, in iteration
    model_output = self.model.forward(input_data, input_lengths)
  File "/mnt/cdtds_cluster_home/s0816700/git/midi_degradation_toolkit/mdtk/pytorch_models.py", line 49, in forward
    self.hidden = self.init_hidden(batch_size, device=device)
  File "/mnt/cdtds_cluster_home/s0816700/git/midi_degradation_toolkit/mdtk/pytorch_models.py", line 40, in init_hidden
    return (torch.randn(1, batch_size, self.hidden_dim, device=device),
RuntimeError: CUDA error: no kernel image is available for execution on the device
</code></pre></div>
<p dir="auto">First tried downgrading to cudatoolkit=10.0, that exhibited same issue.</p>
<p dir="auto">The code will run fine if you repeat steps above but instead <code class="notranslate">conda install pytorch=1.2 cudatoolkit=10.0 -c pytorch</code>.</p>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">If no longer supporting a specific GPU, please bomb out upon load with useful error message.</p>
<h2 dir="auto">Environment</h2>
<p dir="auto">Unfort ran your script after I 'fixed' so pytorch version will be 1.2 here - issue encountered with version 1.3.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Collecting environment information...
PyTorch version: 1.2.0
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Scientific Linux release 7.6 (Nitrogen)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36)
CMake version: version 2.8.12.2

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: Tesla K40m
Nvidia driver version: 430.50
cuDNN version: /usr/lib64/libcudnn.so.6.5.18

Versions of relevant libraries:
[pip3] numpy==1.16.3
[pip3] numpydoc==0.8.0
[conda] blas                      1.0                         mkl  
[conda] mkl                       2019.4                      243  
[conda] mkl-service               2.3.0            py37he904b0f_0  
[conda] mkl_fft                   1.0.15           py37ha843d7b_0  
[conda] mkl_random                1.1.0            py37hd6b4f25_0  
[conda] pytorch                   1.2.0           py3.7_cuda10.0.130_cudnn7.6.2_0    pytorch
[conda] torchvision               0.4.0                py37_cu100    pytorch"><pre class="notranslate"><code class="notranslate">Collecting environment information...
PyTorch version: 1.2.0
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Scientific Linux release 7.6 (Nitrogen)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36)
CMake version: version 2.8.12.2

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: Tesla K40m
Nvidia driver version: 430.50
cuDNN version: /usr/lib64/libcudnn.so.6.5.18

Versions of relevant libraries:
[pip3] numpy==1.16.3
[pip3] numpydoc==0.8.0
[conda] blas                      1.0                         mkl  
[conda] mkl                       2019.4                      243  
[conda] mkl-service               2.3.0            py37he904b0f_0  
[conda] mkl_fft                   1.0.15           py37ha843d7b_0  
[conda] mkl_random                1.1.0            py37hd6b4f25_0  
[conda] pytorch                   1.2.0           py3.7_cuda10.0.130_cudnn7.6.2_0    pytorch
[conda] torchvision               0.4.0                py37_cu100    pytorch
</code></pre></div>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ezyang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ezyang">@ezyang</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gchanan/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gchanan">@gchanan</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zou3519/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zou3519">@zou3519</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jerryzh168/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jerryzh168">@jerryzh168</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ngimel/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ngimel">@ngimel</a></p>