Tensorflow  Does Not Seem to Work With Latest cuDNN 8.8 Version
@tsender  Could you please have a look at [this](https://www.tensorflow.org/install/source#gpu) tested build configuration for different compatible versions and let us know if it helps?
Thank you! Thank you for pointing me to that table. I am currently using cuDNN 8.1 and CUDA 11.2 without any issues. In this past, however, I may have been using a version of cuDNN between 8.1 and 8.7 without problem, hence why I thought that TF would work with any cuDNN 8.x version and submitted this ticket (though, I do not know exactly which version I was using previously as that docker image has been rewritten). @tsender Thank you for the response!
Could you please confirm if the issue resolves in the latest TF version?
Thank you! I'm rather busy at the moment and likely won't be able to get back to this until the end of next week. But if I have time before then I will try. @tsender Could you please let us know if the issue still persists?
Thank you! Hi @sushreebarsa. 

I tried using the tensorflow/tensorflow:2.12.0-gpu docker image, however, it seems to require that I upgrade my cuda version, which I don't have the time to deal with. Seeing that this entire issue was created because I had accidentally created a setup in which TF was not tested/guaranteed to work in, I do not really see a need to keep this ticket open. Once you told me about the driver compatibility page, I had since downgraded back to cuDNN 8.1 and have not had any issues. I also do not have the time to do any further compatibility testing, nor do I even use the latest version of TF in my work. Hence, I think we can close this issue. @tsender Glad it is working for you!
Thank you for your response. Closing this ticket as per your above comment, please feel free to reopen if you still face any issue. Thank you! Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59966">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59966">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

No

### Source

source

### Tensorflow Version

tf 2.8

### Custom Code

Yes

### OS Platform and Distribution

Linux Ubuntu 20.04

### Mobile device

_No response_

### Python version

3.8

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

CUDA 11.2, cuDNN 8.8

### GPU model and memory

_No response_

### Current Behaviour?

```shell
Note: I am using a custom docker container that is based on an older `tensorflow/tensorflow:latest-gpu` image. This image is Ubuntu 20.4 with tf 2.8 and CUDA 11.2. My actual computer running the container is Ubuntu 18.04 with a Quadro RTX 8000 and NVIDIA driver 460.106.0.

I recently upgraded packages in my docker container, and it seems that the newest version of libcudnn8, being cudNN 8.8, conflicts with Tensorflow 2.8 (possibly other versions too, but I can only speak for TF 2.8). I am able to load one of my pre-trained Conv2d-based models just fine, but actually using the model (for both training and prediction) causes trouble. 

As soon as I try to run any data through my model, TF cannot create the cudnn handle and suggests that I have an insufficient driver version. I tried updating the driver version, but that newer version requires CUDA 12.0, which TF is not compatible with. After many hours of trial and error, it seems that if I do NOT upgrade the libcudnn8 library in my docker image, and keep it as the original version of 8.1, then I do not have any problems.

Since someone else reported an issue with cuDNN 8.8 as well on a different matter (https://github.com/tensorflow/tensorflow/issues/59890), it would seem that TF 2 is not compatible with this newer version.
```


### Standalone code to reproduce the issue

```shell
Any piece of code that passes data through a tf.model should result in an error (especially with conv2d layers, since that's what my model uses).
```


### Relevant log output

```shell
2023-03-12 17:40:03.421988: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2023-03-12 17:40:03.422120: E tensorflow/stream_executor/cuda/cuda_dnn.cc:382] Possibly insufficient driver version: 460.106.0
2023-03-12 17:40:03.422182: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at conv_ops.cc:1120 : UNIMPLEMENTED: DNN library is not found.
Traceback (most recent call last):
  File "semantic_segmentation.py", line 378, in <module>
    SemSeg.train()
  File "semantic_segmentation.py", line 231, in train
    loss = self._train_step(image_batch, mask_batch)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 54, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.UnimplementedError: Graph execution error:

Detected at node 'model/conv2d/Conv2D' defined at (most recent call last):
    File "semantic_segmentation.py", line 378, in <module>
      SemSeg.train()
    File "semantic_segmentation.py", line 231, in train
      loss = self._train_step(image_batch, mask_batch)
    File "semantic_segmentation.py", line 213, in _train_step
      predicted_probs = self.model(images, training=True)
    File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/usr/local/lib/python3.8/dist-packages/keras/engine/base_layer.py", line 1096, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/usr/local/lib/python3.8/dist-packages/keras/engine/functional.py", line 451, in call
      return self._run_internal_graph(
    File "/usr/local/lib/python3.8/dist-packages/keras/engine/functional.py", line 589, in _run_internal_graph
      outputs = node.layer(*args, **kwargs)
    File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/usr/local/lib/python3.8/dist-packages/keras/engine/base_layer.py", line 1096, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/usr/local/lib/python3.8/dist-packages/keras/layers/convolutional.py", line 248, in call
      outputs = self.convolution_op(inputs, self.kernel)
    File "/usr/local/lib/python3.8/dist-packages/keras/layers/convolutional.py", line 233, in convolution_op
      return tf.nn.convolution(
Node: 'model/conv2d/Conv2D'
DNN library is not found.
         [[{{node model/conv2d/Conv2D}}]] [Op:__inference__train_step_1535]
```
</details>