Torch doesnt see gpu on gcloud with deep learning containers
I am trying to run some python code on kubernetes in GCloud, I am using pytorch and for the base image i am using gcr.io/deeplearning-platform-release/pytorch-gpu/.
Everything spins up fine, and my model trains but only using CPU. When i run the following
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.cuda.is_available()
/opt/conda/lib/python3.7/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA    initialization: CUDA driver initialization failed, you might not have a CUDA gpu. (Triggered internally at  /opt/conda/conda-bld/pytorch_1614378098133/work/c10/cuda/CUDAFunctions.cpp:109.)
return torch._C._cuda_getDeviceCount() &gt; 0
False

But this is the output of nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.67       Driver Version: 418.67       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |
| N/A   62C    P8    32W / 149W |     11MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

The version of pytorch is the one that is given by the base image, so i am not exactly sure how to fix this. Any help would be appreciated.
EDIT:
&gt;&gt;&gt; torch.__version__
'1.8.0'

EDIT TWO:
&gt;&gt;&gt; torch.version.cuda
'11.1'


The problem is that you have a NVIDIA driver that supports up to CUDA 10.1 and you installed a PyTorch built on CUDA 11.1. To solve that issue you can:

Update your NVIDIA driver to one that supports CUDA 11.1, or
Install a PyTorch compatible with CUDA 10.1 (which is compatible with your NVIDIA driver)

For the option 2, you can simply run the following:
pip install torch==1.8.0+cu101 torchvision==0.9.0+cu101 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html

You can check the versions available for CUDA 10.1 in https://download.pytorch.org/whl/torch_stable.html (those in cu101/)
