TF 2.14 minimum nvidia driver version
@schrummy14,
Could you please confirm if this is the document which you are looking for https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html

and also provide if you are facing any issue/error while the running code mentioned above or the executed smoothly. Thank you! @tilakrayal,
No, not the Nvidia doc. This was part of the tensorflow docs. (`https://www.tensorflow.org/install/pip`)
The code does not complete and errors out.

Below is the full error.
Error occurs for both k80s and v100s.

```txt
2023-10-02 16:01:32.416512: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x31b68fd0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2023-10-02 16:01:32.416555: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla K80, Compute Capability 3.7
2023-10-02 16:01:32.416570: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (1): Tesla K80, Compute Capability 3.7
2023-10-02 16:01:32.416580: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (2): Tesla K80, Compute Capability 3.7
2023-10-02 16:01:32.416590: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (3): Tesla K80, Compute Capability 3.7
2023-10-02 16:01:32.423430: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:447] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2023-10-02 16:01:32.423550: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:451] Memory usage: 466550784 bytes free, 11996954624 bytes total.
2023-10-02 16:01:32.423628: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:461] Possibly insufficient driver version: 450.203.8
2023-10-02 16:01:32.423777: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at xla_ops.cc:624 : FAILED_PRECONDITION: DNN library initialization failed. Look at the errors above for more details.
2023-10-02 16:01:32.426575: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:447] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2023-10-02 16:01:32.426663: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:451] Memory usage: 466550784 bytes free, 11996954624 bytes total.
2023-10-02 16:01:32.426717: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:461] Possibly insufficient driver version: 450.203.8
2023-10-02 16:01:32.426873: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at xla_ops.cc:624 : FAILED_PRECONDITION: DNN library initialization failed. Look at the errors above for more details.
2023-10-02 16:01:32.430015: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:447] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2023-10-02 16:01:32.430104: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:451] Memory usage: 466550784 bytes free, 11996954624 bytes total.
2023-10-02 16:01:32.430150: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:461] Possibly insufficient driver version: 450.203.8
2023-10-02 16:01:32.430300: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at xla_ops.cc:624 : FAILED_PRECONDITION: DNN library initialization failed. Look at the errors above for more details.
2023-10-02 16:01:32.433232: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:447] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2023-10-02 16:01:32.433321: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:451] Memory usage: 466550784 bytes free, 11996954624 bytes total.
2023-10-02 16:01:32.433368: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:461] Possibly insufficient driver version: 450.203.8
2023-10-02 16:01:32.433499: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at xla_ops.cc:624 : FAILED_PRECONDITION: DNN library initialization failed. Look at the errors above for more details.
2023-10-02 16:01:32.437404: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:447] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2023-10-02 16:01:32.437495: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:451] Memory usage: 466550784 bytes free, 11996954624 bytes total.
2023-10-02 16:01:32.437554: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:461] Possibly insufficient driver version: 450.203.8
2023-10-02 16:01:32.437678: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at xla_ops.cc:624 : FAILED_PRECONDITION: DNN library initialization failed. Look at the errors above for more details.
2023-10-02 16:01:32.440590: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:447] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2023-10-02 16:01:32.440676: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:451] Memory usage: 466550784 bytes free, 11996954624 bytes total.
2023-10-02 16:01:32.440731: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:461] Possibly insufficient driver version: 450.203.8
2023-10-02 16:01:32.440884: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at xla_ops.cc:624 : FAILED_PRECONDITION: DNN library initialization failed. Look at the errors above for more details.
---------------------------------------------------------------------------
FailedPreconditionError                   Traceback (most recent call last)
Cell In[6], line 2
      1 # Run GPU Benchmark
----> 2 gpuBench()

Cell In[5], line 8, in gpuBench()
      6 with tf.device('/GPU:0'):
      7     model_gpu = get_model()
----> 8     model_gpu.fit(X_train_scaled, y_train_encoded, epochs = 10)

File /usr/local/lib/python3.11/dist-packages/keras/src/utils/traceback_utils.py:70, in filter_traceback.<locals>.error_handler(*args, **kwargs)
     67     filtered_tb = _process_traceback_frames(e.__traceback__)
     68     # To get the full stack trace, call:
     69     # `tf.debugging.disable_traceback_filtering()`
---> 70     raise e.with_traceback(filtered_tb) from None
     71 finally:
     72     del filtered_tb

File /usr/local/lib/python3.11/dist-packages/tensorflow/python/eager/execute.py:60, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     53   # Convert any objects of type core_types.Tensor to Tensor.
     54   inputs = [
     55       tensor_conversion_registry.convert(t)
     56       if isinstance(t, core_types.Tensor)
     57       else t
     58       for t in inputs
     59   ]
---> 60   tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
     61                                       inputs, attrs, num_outputs)
     62 except core._NotOkStatusException as e:
     63   if name is not None:

FailedPreconditionError: Graph execution error:

Detected at node SGD/StatefulPartitionedCall_4 defined at (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main

  File "<frozen runpy>", line 88, in _run_code

  File "/usr/local/lib/python3.11/dist-packages/ipykernel_launcher.py", line 17, in <module>

  File "/usr/local/lib/python3.11/dist-packages/traitlets/config/application.py", line 1046, in launch_instance

  File "/usr/local/lib/python3.11/dist-packages/ipykernel/kernelapp.py", line 736, in start

  File "/usr/local/lib/python3.11/dist-packages/tornado/platform/asyncio.py", line 195, in start

  File "/usr/lib/python3.11/asyncio/base_events.py", line 604, in run_forever

  File "/usr/lib/python3.11/asyncio/base_events.py", line 1909, in _run_once

  File "/usr/lib/python3.11/asyncio/events.py", line 80, in _run

  File "/usr/local/lib/python3.11/dist-packages/ipykernel/kernelbase.py", line 516, in dispatch_queue

  File "/usr/local/lib/python3.11/dist-packages/ipykernel/kernelbase.py", line 505, in process_one

  File "/usr/local/lib/python3.11/dist-packages/ipykernel/kernelbase.py", line 412, in dispatch_shell

  File "/usr/local/lib/python3.11/dist-packages/ipykernel/kernelbase.py", line 740, in execute_request

  File "/usr/local/lib/python3.11/dist-packages/ipykernel/ipkernel.py", line 422, in do_execute

  File "/usr/local/lib/python3.11/dist-packages/ipykernel/zmqshell.py", line 546, in run_cell

  File "/usr/local/lib/python3.11/dist-packages/IPython/core/interactiveshell.py", line 3024, in run_cell

  File "/usr/local/lib/python3.11/dist-packages/IPython/core/interactiveshell.py", line 3079, in _run_cell

  File "/usr/local/lib/python3.11/dist-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner

  File "/usr/local/lib/python3.11/dist-packages/IPython/core/interactiveshell.py", line 3284, in run_cell_async

  File "/usr/local/lib/python3.11/dist-packages/IPython/core/interactiveshell.py", line 3466, in run_ast_nodes

  File "/usr/local/lib/python3.11/dist-packages/IPython/core/interactiveshell.py", line 3526, in run_code

  File "/tmp/ipykernel_89/3941288529.py", line 2, in <module>

  File "/tmp/ipykernel_89/898971160.py", line 8, in gpuBench

  File "/usr/local/lib/python3.11/dist-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler

  File "/usr/local/lib/python3.11/dist-packages/keras/src/engine/training.py", line 1783, in fit

  File "/usr/local/lib/python3.11/dist-packages/keras/src/engine/training.py", line 1377, in train_function

  File "/usr/local/lib/python3.11/dist-packages/keras/src/engine/training.py", line 1360, in step_function

  File "/usr/local/lib/python3.11/dist-packages/keras/src/engine/training.py", line 1349, in run_step

  File "/usr/local/lib/python3.11/dist-packages/keras/src/engine/training.py", line 1130, in train_step

  File "/usr/local/lib/python3.11/dist-packages/keras/src/optimizers/optimizer.py", line 544, in minimize

  File "/usr/local/lib/python3.11/dist-packages/keras/src/optimizers/optimizer.py", line 1223, in apply_gradients

  File "/usr/local/lib/python3.11/dist-packages/keras/src/optimizers/optimizer.py", line 652, in apply_gradients

  File "/usr/local/lib/python3.11/dist-packages/keras/src/optimizers/optimizer.py", line 1253, in _internal_apply_gradients

  File "/usr/local/lib/python3.11/dist-packages/keras/src/optimizers/optimizer.py", line 1345, in _distributed_apply_gradients_fn

  File "/usr/local/lib/python3.11/dist-packages/keras/src/optimizers/optimizer.py", line 1340, in apply_grad_to_update_var

DNN library initialization failed. Look at the errors above for more details.
	 [[{{node SGD/StatefulPartitionedCall_4}}]] [Op:__inference_train_function_727]
```

nvidia-smi output
```bash
nimbix@jarvice-job-85995-jt7l8:/data$ nvidia-smi 
Wed Oct  4 21:21:57 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.203.08   Driver Version: 450.203.08   CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |
| N/A   33C    P8    27W / 149W |      0MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           Off  | 00000000:00:05.0 Off |                    0 |
| N/A   46C    P8    28W / 149W |      0MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Tesla K80           Off  | 00000000:00:06.0 Off |                    0 |
| N/A   73C    P8    33W / 149W |      0MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  Tesla K80           Off  | 00000000:00:07.0 Off |                    0 |
| N/A   32C    P8    27W / 149W |      0MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
``` Hi, 

For 2.14 version, could you please try with the below configuration.



Version | Python version | Compiler | Build tools | cuDNN | CUDA
-- | -- | -- | -- | -- | --
tensorflow-2.14.0 | 3.9-3.11 | Clang 16.0.0 | Bazel 6.1.0 | 8.7 | 11.8
tensorflow-2.13.0 | 3.8-3.11 | Clang 16.0.0 | Bazel 5.3.0 | 8.6 | 11.8

 This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62032">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62032">No</a>
 will cudnn latest versions work (say 8.9)?

### Issue type

Documentation Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

binary

### TensorFlow version

tf 2.14

### Custom code

No

### OS platform and distribution

Tensorflow Docker image

### Mobile device

_No response_

### Python version

3.11

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

450.203.8

### GPU model and memory

_No response_

### Current behavior?

I have a question on if the minimum nvidia driver version has changed (I believe the current docs state `450.80.02` (https://www.tensorflow.org/install/pip)). The below script ran using the 2.13 docker image. Thank you.

When trying to run a test gpu benchmark, I get the following error:
```text
2023-10-02 16:01:32.433368: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:461] Possibly insufficient driver version: 450.203.8
```


### Standalone code to reproduce the issue

```shell
Below is the code being used:

import tensorflow as tf
from tensorflow import keras
import numpy as np
import matplotlib.pyplot as plt
import timeit

# Download data and scale
(X_train, y_train), (X_test, y_test) = keras.datasets.cifar10.load_data()

# scaling image values between 0-1
X_train_scaled = X_train/255
X_test_scaled = X_test/255

# one hot encoding labels
y_train_encoded = keras.utils.to_categorical(y_train, num_classes = 10, dtype = 'float32')
y_test_encoded = keras.utils.to_categorical(y_test, num_classes = 10, dtype = 'float32')

# Define the model
def get_model():
    model = keras.Sequential([
        keras.layers.Flatten(input_shape=(32,32,3)),
        keras.layers.Dense(3000, activation='relu'),
        keras.layers.Dense(1000, activation='relu'),
        keras.layers.Dense(10, activation='sigmoid')
    ])
    model.compile(optimizer='SGD',
              loss='categorical_crossentropy',
              metrics=['accuracy'])
    return model

# GPU Benchmark
def gpuBench():
    # GPU
    #strategy = tf.distribute.MirroredStrategy()
    #with strategy.scope():
    with tf.device('/GPU:0'):
        model_gpu = get_model()
        model_gpu.fit(X_train_scaled, y_train_encoded, epochs = 10)

gpuBench()
```


### Relevant log output

_No response_