Loaded runtime CuDNN library: 8.5.0 but source was compiled with: 8.6.0. - even in TF 2.12 and last update of cuda drivers
@gg4u I was able to run this official [example code](https://keras.io/examples/generative/vq_vae/) on colab as you can see [here](https://colab.research.google.com/github/keras-team/keras-io/blob/master/examples/generative/ipynb/vq_vae.ipynb). Please make sure that you are using compatible versions for build configurations. Please let us know the cuda version you used ?
Thank you! I got the same error...any solutions yet? I tried to reinstall cudnn, nvidia driver, and cuda but didn't help at all. I am also facing same error tried uninstalling and reinstalling cuda and cudnn, downgraded tensorflow and tried. nothing is working. Please help to close this issue. I am facing this issue from 2 weeks
`Loaded runtime CuDNN library: 8.5.0 but source was compiled with: 8.6.0.  CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library.  If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2023-06-27 12:07:13.521171: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at conv_ops_fused_impl.h:625 : UNIMPLEMENTED: DNN library is not found.` This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60913">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60913">No</a>
 @Gnanendra18 @Jerry-UtopiaCompression Could you please let us know if you are still facing the issue?
Thank you! Hi,
I got the same issue.
The problem was due to the fact the version of the package `nvidia-cudnn-cu11` was 8.5.0.
This package should be upgraded (`pip install -U nvidia-cudnn-cu11`). @znacer that did the trick indeed! I've got 8.6.0 actually installed as a ubuntu package but the python package takes priority. That was an old one and needed updating. I'm in Docker linux and have no idea how to push this forward, I tried upgrade cudnn and tf, downgraded tf, changed docker image from 11 to 12, only thing I found strange, that in TF compatibility matrix there is no CUDNN 8.5 , so have no idea which version of TF i should take. @zba Hi Alexey, 

It is well tricky. What I ended up with is doing this in my `dockerfile`

`RUN pip install -v --default-timeout=120 --no-cache-dir -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cu118`

and this in my `requirements.txt`

`nvidia-cudnn-cu11==8.6.*`

For the `dockerfile` the only thing that matters is (I think)

`RUN pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cu118`

and you could also do

`RUN pip install nvidia-cudnn-cu11==8.6.* --extra-index-url https://download.pytorch.org/whl/cu118`

I hope this works for you. Let us know!

Edit: This was using `FROM tensorflow/tensorflow:2.14.0-gpu` as my base image. I think it was installing PyTorch that bungled it. > RUN pip install nvidia-cudnn-cu11==8.6.* --extra-index-url https://download.pytorch.org/whl/cu118

Hi, thank you, it works, fyi my base image nvidia/cuda:11.8.0-devel-ubuntu22.04  Hi,
I got the same error. 
ubuntu 22.04 , cuda version 12.2 , cudnn 8.8.1, I got the error in tensorflow version 2.15.0 as per documentation cuda 12.2 is supported for tenserflow 2.15.0
I haven't done any changes for cuda version
Just downgraded tenserflow with cuda and cudnn using the command "pip install tensorflow[and-cuda]==2.14.0" so the issue has sorted
 Got the same error with tensorflow 2.15. Fixed it downgrading to 2.14, tf 2.15 installer is broken. It seems that can't recognize nvidia-cudnn-cu11==8.9.4.* installed with pip.
<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

No

### Source

source

### Tensorflow Version

2.12.0

### Custom Code

Yes

### OS Platform and Distribution

Linux Ubuntu 22.

### Mobile device

_No response_

### Python version

_No response_

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

11.8/8

### GPU model and memory

_No response_

### Current Behaviour?

Running a tutorial from Keras official blog with TF 2.12 and with updated drivers in Cuda, but :

```
2023-06-18 11:49:15.576607: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:417] Loaded runtime CuDNN library: 8.5.0 but source was compiled with: 8.6.0.  CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library.  If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2023-06-18 11:49:15.578580: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at conv_ops_fused_impl.h:625 : UNIMPLEMENTED: DNN library is not found.
2023-06-18 11:49:15.578631: I tensorflow/core/common_runtime/executor.cc:1197] [/job:localhost/replica:0/task:0/device:GPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): UNIMPLEMENTED: DNN library is not found.
```

TF and cuda correctly installed as per :
https://www.tensorflow.org/install/pip

I tried to downgrade TF, but won't work because dependencies (tensorflow-probability and protobuf  will fail)



### Standalone code to reproduce the issue

```shell
Cannot reproduce a tutorial from official Keras blog:

https://keras.io/examples/generative/vq_vae/

2023-06-18 11:49:15.576607: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:417] Loaded runtime CuDNN library: 8.5.0 but source was compiled with: 8.6.0.  

I could not solve this error, for TF is updated ( 2.12.0 ) and installed from pip as showed on
https://www.tensorflow.org/install/pip
, cuda as well ypdated. Don't know how to solve:


import tensorflow.python.platform.build_info as build
print(build.build_info)

OrderedDict([('cpu_compiler', '/dt9/usr/bin/gcc'), ('cuda_compute_capabilities', ['sm_35', 'sm_50', 'sm_60', 'sm_70', 'sm_75', 'compute_80']), ('cuda_version', '11.8'), ('cudnn_version', '8'), ('is_cuda_build', True), ('is_rocm_build', False), ('is_tensorrt_build', True)])

```
```


### Relevant log output

```shell
2023-06-18 11:49:15.576607: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:417] Loaded runtime CuDNN library: 8.5.0 but source was compiled with: 8.6.0.  CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library.  If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2023-06-18 11:49:15.578580: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at conv_ops_fused_impl.h:625 : UNIMPLEMENTED: DNN library is not found.
2023-06-18 11:49:15.578631: I tensorflow/core/common_runtime/executor.cc:1197] [/job:localhost/replica:0/task:0/device:GPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): UNIMPLEMENTED: DNN library is not found.
	 [[{{node vq_vae/encoder/conv2d_24/Relu}}]]

---------------------------------------------------------------------------
UnimplementedError                        Traceback (most recent call last)
Cell In[18], line 3
      1 vqvae_trainer = VQVAETrainer(data_variance, latent_dim=16, num_embeddings=128)
      2 vqvae_trainer.compile(optimizer=keras.optimizers.Adam())
----> 3 vqvae_trainer.fit(x_train_scaled, epochs=30, batch_size=128)

File /data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/utils/traceback_utils.py:70, in filter_traceback.<locals>.error_handler(*args, **kwargs)
     67     filtered_tb = _process_traceback_frames(e.__traceback__)
     68     # To get the full stack trace, call:
     69     # `tf.debugging.disable_traceback_filtering()`
---> 70     raise e.with_traceback(filtered_tb) from None
     71 finally:
     72     del filtered_tb

File /data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/tensorflow/python/eager/execute.py:52, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     50 try:
     51   ctx.ensure_initialized()
---> 52   tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
     53                                       inputs, attrs, num_outputs)
     54 except core._NotOkStatusException as e:
     55   if name is not None:

UnimplementedError: Graph execution error:

Detected at node 'vq_vae/encoder/conv2d_24/Relu' defined at (most recent call last):
    File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
      exec(code, run_globals)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/ipykernel_launcher.py", line 17, in <module>
      app.launch_new_instance()
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/traitlets/config/application.py", line 1043, in launch_instance
      app.start()
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 725, in start
      self.io_loop.start()
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 195, in start
      self.asyncio_loop.run_forever()
    File "/usr/lib/python3.10/asyncio/base_events.py", line 600, in run_forever
      self._run_once()
    File "/usr/lib/python3.10/asyncio/base_events.py", line 1896, in _run_once
      handle._run()
    File "/usr/lib/python3.10/asyncio/events.py", line 80, in _run
      self._context.run(self._callback, *self._args)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 513, in dispatch_queue
      await self.process_one()
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 502, in process_one
      await dispatch(*args)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 409, in dispatch_shell
      await result
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 729, in execute_request
      reply_content = await reply_content
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 422, in do_execute
      res = shell.run_cell(
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 540, in run_cell
      return super().run_cell(*args, **kwargs)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3009, in run_cell
      result = self._run_cell(
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3064, in _run_cell
      result = runner(coro)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
      coro.send(None)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3269, in run_cell_async
      has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3448, in run_ast_nodes
      if await self.run_code(code, result, async_=asy):
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3508, in run_code
      exec(code_obj, self.user_global_ns, self.user_ns)
    File "/tmp/ipykernel_953075/2751485705.py", line 3, in <module>
      vqvae_trainer.fit(x_train_scaled, epochs=30, batch_size=128)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/engine/training.py", line 1685, in fit
      tmp_logs = self.train_function(iterator)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/engine/training.py", line 1284, in train_function
      return step_function(self, iterator)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/engine/training.py", line 1268, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/engine/training.py", line 1249, in run_step
      outputs = model.train_step(data)
    File "/tmp/ipykernel_953075/2697647477.py", line 27, in train_step
      reconstructions = self.vqvae(x)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/engine/training.py", line 558, in __call__
      return super().__call__(*args, **kwargs)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/engine/base_layer.py", line 1145, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 96, in error_handler
      return fn(*args, **kwargs)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/engine/functional.py", line 512, in call
      return self._run_internal_graph(inputs, training=training, mask=mask)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/engine/functional.py", line 669, in _run_internal_graph
      outputs = node.layer(*args, **kwargs)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/engine/training.py", line 558, in __call__
      return super().__call__(*args, **kwargs)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/engine/base_layer.py", line 1145, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 96, in error_handler
      return fn(*args, **kwargs)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/engine/functional.py", line 512, in call
      return self._run_internal_graph(inputs, training=training, mask=mask)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/engine/functional.py", line 669, in _run_internal_graph
      outputs = node.layer(*args, **kwargs)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/engine/base_layer.py", line 1145, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 96, in error_handler
      return fn(*args, **kwargs)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/layers/convolutional/base_conv.py", line 321, in call
      return self.activation(outputs)
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/activations.py", line 317, in relu
      return backend.relu(
    File "/data0/home/h21/luas6629/dummy/lib/python3.10/site-packages/keras/backend.py", line 5396, in relu
      x = tf.nn.relu(x)
Node: 'vq_vae/encoder/conv2d_24/Relu'
DNN library is not found.
	 [[{{node vq_vae/encoder/conv2d_24/Relu}}]] [Op:__inference_train_function_4158]
```
</details>