CUDNN failed to allocate the scratch space for the runner
Hi @dark16sider ,

Thanks for reporting issue. Could you please submit a code snippet to replicate and understand the problem. Please confirm whether you are using prebuilt wheel or built from source? If built from source let us know the build command used.

Could you please confirm whether you have cuDNN installed and what version was installed. For TF2.12 version the recommended configurations are listed below.

Version | Python version | Compiler | Build tools | cuDNN | CUDA
-- | -- | -- | -- | -- | --
tensorflow-2.13.0 | 3.8-3.11 | Clang 16.0.0 | Bazel 5.3.0 | 8.6 | 11.8
tensorflow-2.12.0 | 3.8-3.11 | GCC 9.3.1 | Bazel 5.3.0 | 8.6 | 11.8

Also please confirm whether path configuration set for CUDA and cuDNN and confirm the commands for same.

Thanks!
 This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. It seemed like it was a OOM Error for me. Maybe try decreasing the batch size ? Hi @dark16sider ,

Could you please confirm whether above comments worked for you. If you are getting this at the middle of training yeah there is possibility that it might be happening due to OOM problem also. Since you have not pasted any code snippet which may be custom code I would recommend to please check whether unwanted variables that are initialized are getting deleted when there is no more requirement of those ? Also ensure enough memory resources to remove OOM problem from your program.

Providing more details like reproducible code snippet can help us to understand your problem better.

Thanks! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60783">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60783">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug


### Source

source

### Tensorflow Version

Tensorlfow 2.12

### Custom Code

Yes

### OS Platform and Distribution

Ubuntu 22.04.2


### Python version

3.9


### CUDA/cuDNN version

Cuda 11.8,cuDNN  8.6.0

### GPU model and memory

GTX 4070, Vram 11178/12282Mib

### Current Behaviour?

In the middle of training I suddenly get
`Node: 'gradient_tape/model/conv3d_20/Conv3D/Conv3DBackpropFilterV2'
CUDNN failed to allocate the scratch space for the runner or to find a working no-scratch runner.`

I am running 3d Unet segmentation, my dataset is custom generator going through Dataset.I am using multiprocessing. Exact same code and model ran without this error in windows. It was slow so I moved to linux. It is not easy replicating the issue as it sometimes happens so many epochs after.




