Loading cudNN in docker image where tensorflow is installed via pip returns `Error: libnvrtc.so: cannot open shared object file: No such file or directory`
I don't have a solution but just wanted to mention I'm having the same issue with a local install of tensorflow 2.14 via "pip install tensorflow[and-cuda]" on a Debian 12 machine. The solution does seem to be to system install CUDA, but that's not ideal like you say if tensorflow is supposed to be able to install CUDA on it's own. At the very least the current instructions on the [install tensorflow with pip page](https://www.tensorflow.org/install/pip) are not fully correct at the moment with this issue in place. Also seeing this issue. On a system with only the driver installed, `libnvrtc.so` can't be found. 

Edit: Just to add some additional details about my system:
Ubuntu 22.04 with nvidia driver 510 installed using the installer runfile. This is a secure environment that doesn't have access to the nvidia repositories. According to the [2.14.0 release notes](https://github.com/tensorflow/tensorflow/releases/tag/v2.14.0), this driver is all that should be required to install TensorFlow, but it doesn't work due to missing  `libnvrtc.so`. 

Installing CUDA toolkit manually (also using runfile installer) does fix the issue. So I suspect either the docs need updated to say there are additional dependencies or this is a bug where `nvrtc` is not being installed correctly.  @maltel It seems like NVIDIA compiler runtime (NVRTC) is not installed in your Docker image.
To install the prebuilt image, run the following command:
```
docker pull tensorflow/tensorflow:latest-gpu
```
Please let us know if it helps?
Thank you! Same issue on WSL2 Ubuntu 22.04 on Windows 10 using installation with `python3 -m pip install tensorflow[and-cuda]`

On WSL2 Ubuntu 20.04, after upgrading python to 3.10 I did not have this issue on the same Window 10 computer.

 @lightroomstatistics  Thank you for your response here!
Could you please file a new ticket for the issue you are facing?
Thank you! @sushreebarsa I am not using the tensorflow docker image, I am using a custom one (See dockerfile above) so I am not sure pulling the tensorflow docker image is useful. Besides, based on the other comments here it seems like its not an issue with docker, but rather an issue with Ubuntu when only the driver installed.

I don't know what a new ticket is. Is it the same as a new issue?  :-)  @maltel 
Thanks for your above comment.
Sorry for the mistake! I was asking @lightroomstatistics to create a new ticket for WSL2 issue. 
We will dig more into your issue and get back to you soon. Thank you! @maltel Could you try to install NVRTC manually. This can be done by adding the following lines to your Dockerfile:
```
RUN apt-get update && apt-get install -y nvrtc
```
@sachinprasadhs Could you please have a look into this issue!
Thank you! @sushreebarsa I tried installing it manually, but was not successful unfortunately. The package was not found, this is the output;

```
Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package nvrtc
```

Am I doing something wrong?

I found a package called libnvrtc11.2, but it is not the correct version I believe... At least it didnt solve the issue. https://packages.debian.org/bullseye/libs/libnvrtc11.2 There are several different things going on in this issue's comment history.  As to the latter part, the `nvrtc` package is actually called `cuda-nvrtc-X-Y` for whichever specific CUDA X.Y you have (see e.g. https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/ ).  Regarding the original post, the loading of libnvrtc.so instead of libnvrtc.so.XX (via its SONAME) was a bug in that particular build of cuDNN -- this probably will mean you need not only the cuda-nvrtc-11-8 package but also cuda-nvrtc-dev-11-8, so that the libnvrtc.so symlink is made available.
HTH @nkinnaird - for https://github.com/tensorflow/tensorflow/issues/62194#issuecomment-1781418833 , you might want to see https://github.com/tensorflow/tensorflow/issues/61986 .  I think your issue is more likely related to that one than to this one. Thanks for the reply @cliffwoolley! Does that mean there is no way of installing tensorflow and cuda dependencies through pip only for that particular version? (If the TF team doesn't add the cuda-nvrtc-dev-11-8 package?) Based on the comments in ticket #61986 I tried the command mentioned there: 
`python3 -m pip install "tensorflow[and-cuda]==2.15" --extra-index-url https://pypi.nvidia.com` 
and my error `Could not load library libcudnn_cnn_infer.so.8. Error: libnvrtc.so: cannot open shared object file: No such file or directory` didn't occur. So far so good.

My steps:

1. Complete new WSL2 image
2. sudo apt update
3. sudo apt upgrade
4. sudo apt install python3-pip
5. python3 -m pip install --upgrade pip
6. python3 -m pip install "tensorflow[and-cuda]==2.15" --extra-index-url https://pypi.nvidia.com
7. pip install --ignore-installed --no-cache-dir -r requirements_win.txt
8. ran my keras / tensorflow / mflow interference model
### Issue type

Build/Install

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

binary

### TensorFlow version

2.14.0

### Custom code

Yes

### OS platform and distribution

Ubuntu 22.04

### Mobile device

_No response_

### Python version

3.10

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

11.8/8.7

### GPU model and memory

NVIDIA L40, 46BG

### Current behavior?

We are building a Docker image that is running an Ubuntu 22.04. The host machine is also Ubuntu 22.04. We have chosen to not use the prebuilt tensorflow Docker image. We are trying to install tensorflow via pip in the docker image, and at the first glance it seems to work, as `import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))` works perfectly fine and prints something along the lines of `tf.Tensor(-163.40398, shape=(), dtype=float32)`.

However, when we try to load cudNN by running the keras `Model.train_on_batch` method we get the error `Could not load library libcudnn_cnn_infer.so.8. Error: libnvrtc.so: cannot open shared object file: No such file or directory`. Full stack trace is in the "Relevant log ouput" cell.

If we install cuda in the docker image by running the following commands it works though. But we would prefer not installing cuda directly and only through pip.

```
apt-get install wget \
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin \
mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600 \
wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda-repo-ubuntu2204-11-8-local_11.8.0-520.61.05-1_amd64.deb \
dpkg -i cuda-repo-ubuntu2204-11-8-local_11.8.0-520.61.05-1_amd64.deb \
cp /var/cuda-repo-ubuntu2204-11-8-local/cuda-*-keyring.gpg /usr/share/keyrings/ \
apt-get update \
DEBIAN_FRONTEND=noninteractive apt-get -y install cuda \
echo "export PATH=${PATH}:/usr/local/cuda/bin" >> ~/.bashrc \
```

`nvidia-smi` prints

```
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.12             Driver Version: 535.104.12   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA L40                     Off | 00000000:0B:00.0 Off |                    0 |
| N/A   37C    P0              78W / 300W |      4MiB / 46068MiB |      3%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
```

`pip freeze` gives us these dependencies (some dependencies stripped):

```
keras==2.14.0
keras-core==0.1.7
keras-cv==0.6.4
...
numpy==1.26.1
nvidia-cublas-cu11==11.11.3.6
nvidia-cuda-cupti-cu11==11.8.87
nvidia-cuda-nvcc-cu11==11.8.89
nvidia-cuda-runtime-cu11==11.8.89
nvidia-cudnn-cu11==8.7.0.84
nvidia-cufft-cu11==10.9.0.58
nvidia-curand-cu11==10.3.0.86
nvidia-cusolver-cu11==11.4.1.48
nvidia-cusparse-cu11==11.7.5.86
nvidia-nccl-cu11==2.16.5
...
tensorboard==2.14.1
tensorboard-data-server==0.7.1
tensorflow==2.14.0
tensorflow-datasets==4.9.3
tensorflow-estimator==2.14.0
tensorflow-io-gcs-filesystem==0.34.0
tensorflow-metadata==1.14.0
tensorrt==8.5.3.1
```

To me it seems like installing tensorflow via `pip install tensorflow[and-cuda]==2.14.0` doesn't include the libnvrtc.so file? Or maybe it's some error related to keras?


### Standalone code to reproduce the issue

```shell
Dockerfile:


FROM ghcr.io/osgeo/gdal:ubuntu-small-3.7.2

ARG USE_GPU

RUN apt-get update && apt-get install ca-certificates -y

# Install python dependencies
RUN apt install python3-pip -y
RUN pip install pillow==10.1.0 transformers==4.33.3 imageio==2.31.1 scipy==1.11.1 geopandas==0.14.0 dtale==3.7.0 rasterio==1.3.9 rasterstats==0.19.0  psycopg2-binary==2.9.9 sqlalchemy==2.0.22 keras-cv==0.6.4 focal-loss==0.0.7 azure-storage-blob==12.18.3

RUN if [ "$USE_GPU" = "true" ]; then \
    pip install tensorflow[and-cuda]==2.14.0; \
  else \
    pip install tensorflow==2.14.0; \
  fi
```
```


### Relevant log output

```shell
2023-10-23 05:58:21.251995: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-10-23 05:58:21.252097: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-10-23 05:58:21.252135: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-10-23 05:58:21.259174: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Using TensorFlow backend
2023-10-23 05:58:27.246915: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-10-23 05:58:27.284909: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-10-23 05:58:27.286183: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-10-23 05:58:27.298554: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55b5e53fd570 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2023-10-23 05:58:27.298575: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2023-10-23 05:58:27.298922: E ./tensorflow/compiler/xla/stream_executor/stream_executor_internal.h:124] SetPriority unimplemented for this stream.
2023-10-23 05:58:27.299033: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-10-23 05:58:27.300290: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-10-23 05:58:27.301475: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-10-23 05:58:27.447455: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-10-23 05:58:27.448893: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-10-23 05:58:27.450160: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-10-23 05:58:27.451404: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1886] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 43412 MB memory:  -> device: 0, name: NVIDIA L40, pci bus id: 0000:0b:00.0, compute capability: 8.9
2023-10-23 05:58:27.453893: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55b5e5f86f60 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2023-10-23 05:58:27.453916: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA L40, Compute Capability 8.9
learning_rate 0.001
2023-10-23 05:59:11.072927: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:442] Loaded cuDNN version 8700
Could not load library libcudnn_cnn_infer.so.8. Error: libnvrtc.so: cannot open shared object file: No such file or directory
```
