<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">I'm using rtx 3090 to create a simple float32 tensor in cuda. It's very weird it would take too much time after importing huggingface transformers library. However the same code below doesn't have any issues in 20-series cards.</p>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">The following code would reproduce the behavior:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import torch
import time
import transformers

if __name__ == '__main__':
    t = time.time()
    x = torch.zeros((50273,), dtype=torch.float32).cuda()
    print('took:', time.time() - t)
    print('done')"><pre class="notranslate"><code class="notranslate">import torch
import time
import transformers

if __name__ == '__main__':
    t = time.time()
    x = torch.zeros((50273,), dtype=torch.float32).cuda()
    print('took:', time.time() - t)
    print('done')
</code></pre></div>
<p dir="auto">result:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="took: 137.26064133644104
done"><pre class="notranslate"><code class="notranslate">took: 137.26064133644104
done
</code></pre></div>
<p dir="auto">I think the size doesn't matter, any size could reproduce this result in 3090 cards. I don't have rtx 3080, therefore don't know if the issue would exist in 3080 card.</p>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">Here is the result after commenting the <code class="notranslate">import transformers</code>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="took: 0.8175389766693115
done"><pre class="notranslate"><code class="notranslate">took: 0.8175389766693115
done
</code></pre></div>
<h2 dir="auto">Environment</h2>
<p dir="auto">The pytorch is built from the master branch of pytorch github. The transformer library is 3.3.1.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Collecting environment information...
PyTorch version: 1.7.0a0+9d5607f
Is debug build: True
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.1 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-10ubuntu2) 9.3.0
Clang version: Could not collect
CMake version: version 3.18.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce RTX 3090
Nvidia driver version: 455.23.05
cuDNN version: Probably one of the following:
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn.so.8
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_train.so.8
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_train.so.8
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.1
[pip3] numpydoc==1.1.0
[pip3] pytorch-pretrained-bert==0.6.2
[pip3] torch==1.7.0a0
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               10.2.89              hfd86e86_1  
[conda] mkl                       2020.2                      256  
[conda] mkl-include               2020.2                      256  
[conda] mkl-service               2.3.0            py38he904b0f_0  
[conda] mkl_fft                   1.1.0            py38h23d657b_0  
[conda] mkl_random                1.1.1            py38h0573a6f_0  
[conda] numpy                     1.18.5                   pypi_0    pypi
[conda] numpy-base                1.19.1           py38hfa32c7d_0  
[conda] numpydoc                  1.1.0                      py_0  
[conda] pytorch-pretrained-bert   0.6.2                    pypi_0    pypi
[conda] torch                     1.7.0a0                  pypi_0    pypi"><pre class="notranslate"><code class="notranslate">Collecting environment information...
PyTorch version: 1.7.0a0+9d5607f
Is debug build: True
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.1 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-10ubuntu2) 9.3.0
Clang version: Could not collect
CMake version: version 3.18.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce RTX 3090
Nvidia driver version: 455.23.05
cuDNN version: Probably one of the following:
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn.so.8
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_train.so.8
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_train.so.8
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.1
[pip3] numpydoc==1.1.0
[pip3] pytorch-pretrained-bert==0.6.2
[pip3] torch==1.7.0a0
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               10.2.89              hfd86e86_1  
[conda] mkl                       2020.2                      256  
[conda] mkl-include               2020.2                      256  
[conda] mkl-service               2.3.0            py38he904b0f_0  
[conda] mkl_fft                   1.1.0            py38h23d657b_0  
[conda] mkl_random                1.1.1            py38h0573a6f_0  
[conda] numpy                     1.18.5                   pypi_0    pypi
[conda] numpy-base                1.19.1           py38hfa32c7d_0  
[conda] numpydoc                  1.1.0                      py_0  
[conda] pytorch-pretrained-bert   0.6.2                    pypi_0    pypi
[conda] torch                     1.7.0a0                  pypi_0    pypi
</code></pre></div>
<h2 dir="auto">Additional context</h2>
<p dir="auto">I don't know the issue come from transformer library or pytorch. But since the same code under same setting works in rtx 2080. I guess there is a higher chance the problem come from pytorch itself. Thanks for any help.</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ngimel/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ngimel">@ngimel</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/csarofeen/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/csarofeen">@csarofeen</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ptrblck/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ptrblck">@ptrblck</a></p>
<h2 dir="auto">Edit: If you see this error, please go to <a href="https://pytorch.org/get-started/locally/" rel="nofollow">https://pytorch.org/get-started/locally/</a> and download a wheel built with cuda 11.x</h2>
<hr>
<p dir="auto">**Hi</p>
<h1 dir="auto">I recently purchased RTX 3080 and got this error</h1>
<h1 dir="auto">GeForce RTX 3080 with CUDA capability sm_86 is not compatible with the current PyTorch installation.<br>
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_61 sm_70 sm_75 compute_37.</h1>
<p dir="auto">and my cuda version is following:</p>
<ul dir="auto">
<li>torch.cuda_version</li>
<li>Out[3]: '10.2'</li>
</ul>
<p dir="auto">Should I install the latest cudatoolkit 11.0? but It seems Pytorch only provides cudatoolkit 10.2 like below screenshot.<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/52276191/93667640-7ca4dc80-fac2-11ea-80de-47cbdcfa9cd5.png"><img src="https://user-images.githubusercontent.com/52276191/93667640-7ca4dc80-fac2-11ea-80de-47cbdcfa9cd5.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto">Is there any solution for this issue?<br>
How do I get sm_86 binary capability?</p>
<p dir="auto">thanks in advance**</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/malfet/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/malfet">@malfet</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/seemethere/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/seemethere">@seemethere</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/walterddr/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/walterddr">@walterddr</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ngimel/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ngimel">@ngimel</a></p>