TensorFlow Cuda in Docker under WSL2 not wokring
Based on the log output and the steps you’ve taken, it appears that TensorFlow within your Docker container is unable to initialize CUDA, leading to the error `CUDA_ERROR_NOT_FOUND: named symbol not found`. This usually indicates an issue with CUDA installation or GPU driver compatibility within the Docker environment.

Here's a standalone code snippet and Docker run command to reproduce the issue, as well as additional steps to troubleshoot and resolve it.

### Standalone Code to Reproduce the Issue

```sh
docker run --rm --gpus all -it tensorflow/tensorflow:latest-gpu bash -c "python3 -c 'import tensorflow as tf; print(tf.config.list_physical_devices(\"GPU\"))'"
```

### Relevant Log Output

```plaintext
2024-05-27 18:05:30.149964: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-05-27 18:05:31.089452: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:282] failed call to cuInit: CUDA_ERROR_NOT_FOUND: named symbol not found
[]
```

### Troubleshooting Steps

#### 1. Verify GPU Driver and CUDA Toolkit in WSL2

Ensure that the GPU driver and CUDA toolkit are correctly installed in your WSL2 environment.

1. **Install NVIDIA Drivers**:

   Follow the [NVIDIA guide for WSL2](https://developer.nvidia.com/cuda/wsl) to install the latest GPU driver for WSL2.

2. **Install CUDA Toolkit in WSL2**:

   Install the CUDA toolkit by following the official [NVIDIA CUDA on WSL user guide](https://docs.nvidia.com/cuda/wsl-user-guide/index.html).

   ```sh
   sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub
   sudo sh -c 'echo "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 /" > /etc/apt/sources.list.d/cuda.list'
   sudo apt-get update
   sudo apt-get -y install cuda
   ```

#### 2. Check Docker Configuration

Ensure Docker is configured to use the NVIDIA runtime.

1. **Install NVIDIA Container Toolkit**:

   ```sh
   sudo apt-get install -y nvidia-docker2
   sudo systemctl restart docker
   ```

2. **Set Default Runtime**:

   Add the following to `/etc/docker/daemon.json`:

   ```json
   {
     "runtimes": {
       "nvidia": {
         "path": "nvidia-container-runtime",
         "runtimeArgs": []
       }
     }
   }
   ```

   Restart Docker:

   ```sh
   sudo systemctl restart docker
   ```

#### 3. Verify CUDA in Docker Container

Ensure that the Docker container can access CUDA.

1. **Run CUDA Container**:

   ```sh
   docker run --rm --gpus all nvidia/cuda:11.8-base nvidia-smi
   ```

2. **Test CUDA Program**:

   Run a simple CUDA program inside the TensorFlow Docker container.

   ```sh
   docker run --rm --gpus all -it tensorflow/tensorflow:latest-gpu bash -c "apt-get update && apt-get install -y cuda-samples-11-8 && cd /usr/local/cuda-11.8/samples/1_Utilities/deviceQuery && make && ./deviceQuery"
   ```

#### 4. Set Environment Variables in Docker Container

Ensure the correct CUDA environment variables are set within the container.

1. **Run TensorFlow with CUDA Environment Variables**:

   ```sh
   docker run --rm --gpus all -it tensorflow/tensorflow:latest-gpu bash -c "export LD_LIBRARY_PATH=/usr/local/cuda/lib64 && export CUDA_HOME=/usr/local/cuda && python3 -c 'import tensorflow as tf; print(tf.config.list_physical_devices(\"GPU\"))'"
   ```

#### 5. Reinstall GPU Drivers on Host

Reinstall the NVIDIA GPU drivers on your host machine to ensure compatibility with WSL2 and Docker.

1. **Clean Installation of NVIDIA Drivers**:

   - Use the [Display Driver Uninstaller (DDU)](https://www.guru3d.com/files-details/display-driver-uninstaller-download.html) to uninstall existing drivers.
   - Download and install the latest drivers from the [NVIDIA website](https://www.nvidia.com/Download/index.aspx).

#### 6. Update Docker and WSL2

Ensure Docker Desktop and WSL2 are up to date.

1. **Update Docker Desktop**:

   Download and install the latest version of Docker Desktop from the [Docker website](https://www.docker.com/products/docker-desktop).

2. **Update WSL2 Kernel**:

   Update the WSL2 kernel:

   ```sh
   wsl --update
   ```
 You do not need the Nvidia Container Toolkit in WSL. It's only necessary on a native Ubuntu host. I have tried with and without and it doesn't make a difference. @koryphaee I have solved the issue by installing an older NVIDIA driver (552.44). I have installed it with the "Perform a clean install" option from the Custom (Advanced) installation menu. You can find older versions here: [https://www.nvidia.com/Download/Find.aspx#](https://www.nvidia.com/Download/Find.aspx#)

![Untitled](https://github.com/tensorflow/tensorflow/assets/13253183/46d3947e-c991-4346-834a-c6f47e9df3e5)

The only difference is downgrading the driver version. @nikolayDemirev,
I was successfully able to use tensorflow-gpu with docker with Tensorflow 2.16.1. Could you please check if the requirements are installed properly or not.

https://stackoverflow.com/questions/78418499/using-tensorflow-with-gpu-on-docker-on-ubuntu

![image](https://github.com/tensorflow/tensorflow/assets/81610181/6ce34499-4ee4-4ba3-bb63-a3f3425768a7)
![image](https://github.com/tensorflow/tensorflow/assets/81610181/88b5be01-c89e-447c-bd13-42e93559260b)
 > @nikolayDemirev, I was successfully able to use tensorflow-gpu with docker with Tensorflow 2.16.1. Could you please check if the requirements are installed properly or not.
> 
> https://stackoverflow.com/questions/78418499/using-tensorflow-with-gpu-on-docker-on-ubuntu
> 
> ![image](https://private-user-images.githubusercontent.com/81610181/334478638-6ce34499-4ee4-4ba3-bb63-a3f3425768a7.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTY5MTA3NTMsIm5iZiI6MTcxNjkxMDQ1MywicGF0aCI6Ii84MTYxMDE4MS8zMzQ0Nzg2MzgtNmNlMzQ0OTktNGVlNC00YmEzLWJiNjMtYTNmMzQyNTc2OGE3LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA1MjglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNTI4VDE1MzQxM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWU0ZmVmOGVkYWZmMjc3NTE5MjA1MjIxMjJhYWYyNzRhZTcxMzY4YzVlMjk0N2FkNTVlOGVjZTc2MmYxMDI4YjEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.XjWYDWqjhpd47LAaYn3ESERz-7myP9HgYhmI9XsIKJI) ![image](https://private-user-images.githubusercontent.com/81610181/334478794-88b5be01-c89e-447c-bd13-42e93559260b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTY5MTA3NTMsIm5iZiI6MTcxNjkxMDQ1MywicGF0aCI6Ii84MTYxMDE4MS8zMzQ0Nzg3OTQtODhiNWJlMDEtYzg5ZS00NDdjLWJkMTMtNDJlOTM1NTkyNjBiLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA1MjglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNTI4VDE1MzQxM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWNiZWUwNWYzZGMyZmU1ZmIxYTM5MGFiZGU1OTZkMTM1NGE4ZjU0YWEzMjllODExMTgyMTgzZmFmZGEwZDE2NzcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.CXI5xAExkyIQMN9XJTtE8TmOhL7b2bs3e_e6JuFuL2E)

I have only gotten it working by downgrading the Nvidia driver to version 552.44. Nothing else helped. It seems the latest GPU driver, 555.85, has some issues. I can confirm the workaround suggested by @nikolayDemirev is working. After downgrading from driver version 555.85 to 552.44 TensorFlow successfully detects the GPU with CUDA.
Since this appears to be a regression in the NVidia driver it would probably be best to lets them know. Is there somebody at TensorFlow who can do that? I could contact customer support but I'm not sure I would reach the right people. The solution works for me...

Num GPUs Available:  1 > I can confirm the workaround suggested by @nikolayDemirev is working. After downgrading from driver version 555.85 to 552.44 TensorFlow successfully detects the GPU with CUDA. Since this appears to be a regression in the NVidia driver it would probably be best to lets them know. Is there somebody at TensorFlow who can do that? I could contact customer support but I'm not sure I would reach the right people.

I had the same problem too, 555 detects GPU with nvidia-smi but does not pass it to TensorFlow. I have downgraded to 552 and now TensorFlow detects the GPU and uses it.
Thank you :) @nikolayDemirev , @koryphaee ,
Looks like the issue is from the Nvidia GPU driver, where it is working with the driver version **552.44 or below**. And on Nvidia driver and the tool kit, we don't have control to make the changes on the same. Could you please try to raise the concern on the Nvidia forum for the quick resolution. Thank you! The relevant forums seems to be https://forums.developer.nvidia.com/c/developer-tools/cuda-developer-tools/285. Sadly I am not allowed to post there (I don't know why). I have submitted a support ticket instead and linked this issue. I will let you know if they respond. Their support redirected me back to a more specific forum and I made a post there: https://forums.developer.nvidia.com/t/driver-555-85-is-unable-to-detect-gpu/294495 NVIDIA is working on this.  If you user Docker CE on the Linux side of WSL, then update your nvidia-container-toolkit to 1.14.4 or later.  If you are using Docker Desktop, then for now the best approach is to use a driver 552.xx or earlier.  I will post back here as soon as the remaining incompatibility is resolved.

Cliff Woolley
Sr. Director, Deep Learning Software
NVIDIA We will track this in https://github.com/NVIDIA/nvidia-container-toolkit/issues/520 going forward, thanks. [Docker Desktop 4.31](https://docs.docker.com/desktop/release-notes/#4310) was released yesterday and includes NVIDIA Container Toolkit 1.15.0, which resolves this issue. I can confirm it works now using
* Docker Desktop 4.31
* GPU Driver 555.99

Running
```
docker run --rm --gpus all -it tensorflow/tensorflow:latest-gpu python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
```
yields
```
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
```

Thank you for the quick fix and letting us know <3 @nikolayDemirev @koryphaee
Hope this issue has been fixed from the Nvidia side with the mentioned issue https://github.com/NVIDIA/nvidia-container-toolkit/issues/520


Could you please feel free to move this issue to the closed status. Thank you!
### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

binary

### TensorFlow version

2.16.1-gpu-jupyter

### Custom code

No

### OS platform and distribution

Windows 11 and WSL2

### Mobile device

_No response_

### Python version

_No response_

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

release 10.1, V10.1.243

### GPU model and memory

_No response_

### Current behavior?

I installed the latest NVIDIA driver on Windows 11 and configured the Docker Desktop to run with WSL2. On Ubuntu, I have installed the Nvidia Container Toolkit, following the instructions not to install drivers on Linux.

nvidia-smi is showing me the following output in WSL:
![wls-smi](https://github.com/tensorflow/tensorflow/assets/13253183/2aa35674-3ca8-4f0e-b505-bd134a4a1e06)

Running a docker container with the following command:
```
sudo docker run -it --rm -p 8888:8888 --gpus all tensorflow/tensorflow:latest-gpu-jupyter
```
And running:
```
import tensorflow as tf
tf.config.list_physical_devices()
```
results in the following error "external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:282] failed call to cuInit: CUDA_ERROR_NOT_FOUND: named symbol not found":
![jupyter](https://github.com/tensorflow/tensorflow/assets/13253183/3104b88b-953a-401b-8658-5b31e0c07cc5)

Running nvidia-smi in the same container results in the same result as running in on WLS.

### Standalone code to reproduce the issue

```shell
You only need to run

sudo docker run -it --rm -p 8888:8888 --gpus all tensorflow/tensorflow:latest-gpu-jupyter
```
in WSL2 and then run:
```
import tensorflow as tf
tf.config.list_physical_devices()
```
```


### Relevant log output

```shell
2024-05-27 18:02:30.907995: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-05-27 18:02:31.117531: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-05-27 18:02:32.496598: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:282] failed call to cuInit: CUDA_ERROR_NOT_FOUND: named symbol not found
```
