WSL2 Pytorch - RuntimeError: No CUDA GPUs are available with RTX3080
I have been struggling for day to make torch work on WSL2 using an RTX 3080.
I Installed the CUDA-toolkit version 11.3
Running nvcc -V returns this :
nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Mar_21_19:15:46_PDT_2021
Cuda compilation tools, release 11.3, V11.3.58
Build cuda_11.3.r11.3/compiler.29745058_0

nvidia-smi returns this
Mon Nov 29 00:38:26 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.00       Driver Version: 510.06       CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:01:00.0  On |                  N/A |
| N/A   52C    P5    17W /  N/A |   1082MiB / 16384MiB |     N/A      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

I verified the installation of the toolkit with blackscholes
./BlackScholes
[./BlackScholes] - Starting...
GPU Device 0: &quot;Ampere&quot; with compute capability 8.6

Initializing data...
...allocating CPU memory for options.
...allocating GPU memory for options.
...generating input data in CPU mem.
...copying input data to GPU mem.
Data init done.

Executing Black-Scholes GPU kernel (512 iterations)...
Options count             : 8000000
BlackScholesGPU() time    : 0.242822 msec
Effective memory bandwidth: 329.459087 GB/s
Gigaoptions per second    : 32.945909

BlackScholes, Throughput = 32.9459 GOptions/s, Time = 0.00024 s, Size = 8000000 options, NumDevsUsed = 1, Workgroup = 128

Reading back GPU results...
Checking the results...
...running CPU calculations.

Comparing the results...
L1 norm: 1.741792E-07
Max absolute error: 1.192093E-05

Shutting down...
...releasing GPU memory.
...releasing CPU memory.
Shutdown done.

[BlackScholes] - Test Summary

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

Test passed

And when I try to use torch, it doesn't find any GPU. Btw, I had to install torch==1.10.0+cu113 if I wanted to use torch with my RTX 3080 as the sm_ with the simple 1.10.0 version are not compatible with the rtx3080.
Running torch returns this :
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.version
&lt;module 'torch.version' from '/home/snihar/miniconda3/envs/tscbasset/lib/python3.7/site-packages/torch/version.py'&gt;
&gt;&gt;&gt; torch.version.cuda
'11.3'
&gt;&gt;&gt; torch.cuda.get_arch_list()
[]
&gt;&gt;&gt; torch.cuda.device_count()
0
&gt;&gt;&gt;  torch.cuda.current_device()
Traceback (most recent call last):
  File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
  File &quot;/home/snihar/miniconda3/envs/tscbasset/lib/python3.7/site-packages/torch/cuda/__init__.py&quot;, line 479, in current_device
    _lazy_init()
  File &quot;/home/snihar/miniconda3/envs/tscbasset/lib/python3.7/site-packages/torch/cuda/__init__.py&quot;, line 214, in _lazy_init
    torch._C._cuda_init()
RuntimeError: No CUDA GPUs are available

At last, interestingly, I am completely able to run tensorflow-gpu on the same machine.
Installed pytorch like this : conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
Also, I managed to run pytorch in a docker container started from my WSL2 machine with this command :
sudo docker run --gpus all -it --rm -v /home/...:/home/... nvcr.io/nvidia/pytorch:21.11-py3.  

When running pytorch on the windows machine I am running the WSL from, it works too. Both return  ['sm_37', 'sm_50', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'compute_37'] which says that the library is compatible with rtx 3080.

In my case, I solved this issue by linking /usr/lib/wsl/lib/libcuda.so.1 to the libcuda.so in your wsl2 CUDA location. See https://github.com/microsoft/WSL/issues/5663
After reboot, pytorch can find the GPU.
(I found the warning &quot; /usr/lib/wsl/lib/libcuda.so.1 is not a symbolic link&quot; during the apt-get upgrade command. Not sure you can solve it in the same way) Downgrade to pytorch 1.8.2LTS can also solve the problem, but the calculation speed is extremely low.
