GKE - GPU nvidia - cuda drivers dont work I have setup a kubernetes node with a Nvidia Tesla k80 and followed this tutorial to try to run a pytorch docker image with Nvidia drivers and CUDA drivers working.

I have managed to install the nvidia daemonsets and I can now see the following pods: 

nvidia-driver-installer-gmvgt
nvidia-gpu-device-plugin-lmj84


The problem is that even while using the recommended image nvidia/cuda:10.0-runtime-ubuntu18.04 I still can't find the Nvidia drivers inside my pod:

root@pod-name-5f6f776c77-87qgq:/app# ls /usr/local/
bin  cuda  cuda-10.0  etc  games  include  lib  man  sbin  share  src


But the tutorial mentions:


  CUDA libraries and debug utilities are made available inside the container at /usr/local/nvidia/lib64 and /usr/local/nvidia/bin, respectively.


I have also tried to test if CUDA was working through torch.cuda.is_available() but I get False as a return value.

Many thanks in advance for your help.
Ok so i finally made it work.

It is mandatory set a ressource limit to access the nvidia driver, which is weird considering either way my pod was on the right node with the nvidia drivers installed..