Error to train or load data on RTX 4080
@gaikwadrahul8 
I tried to replicate the issue on wsl2 using tf-nightly but facing a different error . Please refer to  the following screenshot and output log for reference:

![wsl-test1](https://user-images.githubusercontent.com/111861663/224571815-a5f25190-8fd7-4458-8ef4-2a9c46108df0.png)
**Output log**-[output_log](https://github.com/tensorflow/tensorflow/files/10952017/tf.root%40MoRaInE_.python.00_cifar.txt)


Could you please look into this ?

Thank you ! Hello! Sorry i forget to comment that I found the problem. The new 40 series GPU needs CUDA 11.8 or higher and cuDNN 8.6 or higher. And that versions aren't available on Conda, so you need to install on system-side or globally  Hi, @enmanuelmag 

Apologize for the delayed response and if you're using `Nvidia GPU RTX 4080 ` GPU then you'll have use `cuda==11.8` and `cuDNN==8.6` as per [official Nvidia developer forum ](https://forums.developer.nvidia.com/t/tensorflow-and-cuda-12/237333/5?u=gaikwadrahul) and at the moment you're using `cuda==11.2.0` and `cuDNN==8.1.0 `  so I would request you to please give it try with `cuda==11.8` and `cuDNN==8.6` and check whether is it working as expected ? you can also refer our official [tested build configuration ](https://www.tensorflow.org/install/source#tested_build_configurations) section.

If you still persists issue please let us know or Could you please confirm if this issue is resolved for you ? Please feel free to close the issue if it is resolved ? Thank you!
 This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59897">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59897">No</a>
 nice issues
<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

Yes

### Source

source

### Tensorflow Version

tf 2.13

### Custom Code

Yes

### OS Platform and Distribution

WSL 2

### Mobile device

22.04

### Python version

3.9

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

11.2.0/8.1.0

### GPU model and memory

RTX 4080

### Current Behaviour?

```shell
The error occurs at the time of performing the workouts or loading data with the tf.Dataset API.

At first I tried with my native Windows environment, I followed the installation instructions on the tensorflow page but I got an unexplainable error, the kernel just died.

Finally I tried with the nightly version and the error I have now is supposedly a memory error, which seems strange to me since I tried downloading the batch and nothing. Then I tried running a very small bechmark that I have run with my previous GPU and I get the same error (the benchmark is this one: https://github.com/mrdbourke/m1-machine-learning-test).
```


### Standalone code to reproduce the issue

```shell
GPU RTX 4080
Tensorflow nightly (2.13)
Jupyter notebook to execute: https://github.com/mrdbourke/m1-machine-learning-test/blob/main/00_cifar10_tinyvgg_benchmark.ipynb
```


### Relevant log output

```shell
This output error was running o windows native (tensorflow 2.10): kernel just died


info 16:03:03.770: Got new session 6aead894-0d87-44cf-ba16-f4a714a3ce8f
info 16:03:03.770: Started new restart session
info 16:03:03.777: Dispose Kernel process 14984.
warn 16:03:10.821: StdErr from Kernel Process 2023-03-04 16:03:10.821877: I tensorflow/
warn 16:03:10.821: StdErr from Kernel Process core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

warn 16:03:11.207: StdErr from Kernel Process 2023-03-04 16:03:11.207450: I tensorflow
warn 16:03:11.207: StdErr from Kernel Process /core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13402 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 4080, pci bus id: 0000:08:00.0, compute capability: 8.9

error 16:03:11.949: Disposing session as kernel process died ExitCode: 3221226505, Reason: c:\Users\emman\anaconda3\envs\tf-env\lib\site-packages\traitlets\traitlets.py:2548: FutureWarning: Supporting extra quotes around strings is deprecated in traitlets 5.0. You can use 'hmac-sha256' instead of '"hmac-sha256"' if you require traitlets >=5.
  warn(
c:\Users\emman\anaconda3\envs\tf-env\lib\site-packages\traitlets\traitlets.py:2499: FutureWarning: Supporting extra quotes around Bytes is deprecated in traitlets 5.0. Use '17823775-70bc-4db8-8377-3a19f2263309' instead of 'b"17823775-70bc-4db8-8377-3a19f2263309"'.
  warn(
2023-03-04 16:03:10.821877: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-03-04 16:03:11.207450: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13402 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 4080, pci bus id: 0000:08:00.0, compute capability: 8.9

info 16:03:11.950: Dispose Kernel process 20040.
error 16:03:11.951: Raw kernel process exited code: 3221226505
error 16:03:11.954: Error in waiting for cell to complete Error: Canceled future for execute_request message before replies were done
    at t.KernelShellFutureHandler.dispose (c:\Users\emman\.vscode\extensions\ms-toolsai.jupyter-2023.2.1000592019\out\extension.node.js:2:33213)
    at c:\Users\emman\.vscode\extensions\ms-toolsai.jupyter-2023.2.1000592019\out\extension.node.js:2:52265
    at Map.forEach (<anonymous>)
    at y._clearKernelState (c:\Users\emman\.vscode\extensions\ms-toolsai.jupyter-2023.2.1000592019\out\extension.node.js:2:52250)
    at y.dispose (c:\Users\emman\.vscode\extensions\ms-toolsai.jupyter-2023.2.1000592019\out\extension.node.js:2:45732)
    at c:\Users\emman\.vscode\extensions\ms-toolsai.jupyter-2023.2.1000592019\out\extension.node.js:17:127079
    at ee (c:\Users\emman\.vscode\extensions\ms-toolsai.jupyter-2023.2.1000592019\out\extension.node.js:2:1552543)
    at lh.dispose (c:\Users\emman\.vscode\extensions\ms-toolsai.jupyter-2023.2.1000592019\out\extension.node.js:17:127055)
    at ph.dispose (c:\Users\emman\.vscode\extensions\ms-toolsai.jupyter-2023.2.1000592019\out\extension.node.js:17:134354)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
warn 16:03:11.956: Cell completed with errors {
  message: 'Canceled future for execute_request message before replies were done'
}
info 16:03:11.959: Cancel all remaining cells true || Error || undefined
```


This output error was running on WSL 2 (tensorflow nightly 2.13): ail to allocate memory even when the GPU memory is free (16 GB)
```
2023-03-04 22:55:38.084081: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-03-04 22:55:38.084850: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:967] could not open file to read NUMA node: /sys/bus/pci/devices/0000:08:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-03-04 22:55:38.084914: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:967] could not open file to read NUMA node: /sys/bus/pci/devices/0000:08:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-03-04 22:55:38.084958: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:967] could not open file to read NUMA node: /sys/bus/pci/devices/0000:08:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-03-04 22:55:38.807671: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:967] could not open file to read NUMA node: /sys/bus/pci/devices/0000:08:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-03-04 22:55:38.808089: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:967] could not open file to read NUMA node: /sys/bus/pci/devices/0000:08:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-03-04 22:55:38.808116: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1700] Could not identify NUMA node of platform GPU id 0, defaulting to 0.  Your kernel may not have been built with NUMA support.
2023-03-04 22:55:38.808171: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:967] could not open file to read NUMA node: /sys/bus/pci/devices/0000:08:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-03-04 22:55:38.808390: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13344 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 4080, pci bus id: 0000:08:00.0, compute capability: 8.9
2023-03-04 22:55:39.164350: W tensorflow/tsl/framework/cpu_allocator_impl.cc:82] Allocation of 1228800000 exceeds 10% of free system memory.
2023-03-04 22:55:39.904405: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:735] failed to allocate 13.03G (13992198144 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-03-04 22:55:40.043405: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:735] failed to allocate 11.73G (12592977920 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-03-04 22:55:40.185360: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:735] failed to allocate 10.55G (11333680128 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-03-04 22:55:40.328259: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:735] failed to allocate 9.50G (10200311808 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-03-04 22:55:40.651989: W tensorflow/tsl/framework/cpu_allocator_impl.cc:82] Allocation of 1228800000 exceeds 10% of free system memory.
```
</details>