During trainning model with gpu,  randomly stucked and got windows bluescreen crash
The memory and video memory did not overflow, Windows Task Manager shows that the trend of memory and video memory is very stable.(about 60% of the max, I guess? it shows a straight line.) I checked the dump file with WinDbg.
KEY_VALUES_STRING: 1

    Key  : Analysis.CPU.mSec
    Value: 3015

    Key  : Analysis.DebugAnalysisManager
    Value: Create

    Key  : Analysis.Elapsed.mSec
    Value: 124272

    Key  : Analysis.Init.CPU.mSec
    Value: 406

    Key  : Analysis.Init.Elapsed.mSec
    Value: 56593

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 82

    Key  : WER.OS.Branch
    Value: vb_release

    Key  : WER.OS.Timestamp
    Value: 2019-12-06T14:06:00Z

    Key  : WER.OS.Version
    Value: 10.0.19041.1


BUGCHECK_CODE:  133

BUGCHECK_P1: 1

BUGCHECK_P2: 1e00

BUGCHECK_P3: fffff8024c8fb320

BUGCHECK_P4: 0

DPC_TIMEOUT_TYPE:  DPC_QUEUE_EXECUTION_TIMEOUT_EXCEEDED

TRAP_FRAME:  fffff80251b19310 -- (.trap 0xfffff80251b19310)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=0000000000000000 rbx=0000000000000000 rcx=fffff8026cadbab8
rdx=fffff80251b195e0 rsi=0000000000000000 rdi=0000000000000000
rip=fffff8026bfa2b28 rsp=fffff80251b194a8 rbp=fffff80251b195e0
 r8=fffff80251b19630  r9=000000000000001f r10=00000000ffffffff
r11=000000000000000c r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei ng nz na pe nc
nvlddmkm+0x192b28:
fffff802`6bfa2b28 4053            push    rbx
Resetting default scope

CUSTOMER_CRASH_COUNT:  1

PROCESS_NAME:  System

STACK_TEXT:  
fffff802`51b28e18 fffff802`4c037996     : 00000000`00000133 00000000`00000001 00000000`00001e00 fffff802`4c8fb320 : nt!KeBugCheckEx
fffff802`51b28e20 fffff802`4be539e3     : 0000024f`dd929daf fffff802`490b4180 00000000`00000000 fffff802`490b4180 : nt!KeAccumulateTicks+0x1e1756
fffff802`51b28e80 fffff802`4be534ca     : fffff802`4c8f3940 fffff802`51b19390 fffff802`4ab81500 00000000`00008101 : nt!KeClockInterruptNotify+0x453
fffff802`51b28f30 fffff802`4bf00825     : fffff802`4c8f3940 fffff802`51b28f40 00000000`00000010 ffff6fa3`9b8ce98a : nt!HalpTimerClockIpiRoutine+0x1a
fffff802`51b28f60 fffff802`4bfff6da     : fffff802`51b19390 fffff802`4c8f3940 fffff802`51b195e0 00000000`00000000 : nt!KiCallInterruptServiceRoutine+0xa5
fffff802`51b28fb0 fffff802`4bfffee7     : 00000000`00000000 fffff802`6cadbac0 fffff802`51b19650 fffff802`6cadbac0 : nt!KiInterruptSubDispatchNoLockNoEtw+0xfa
fffff802`51b19310 fffff802`6bfa2b28     : fffff802`6bebf451 fffff802`51b19630 ffff980c`cd530000 ffffc7ad`1671fa95 : nt!KiInterruptDispatchNoLockNoEtw+0x37
fffff802`51b194a8 fffff802`6bebf451     : fffff802`51b19630 ffff980c`cd530000 ffffc7ad`1671fa95 ffff980c`cd530000 : nvlddmkm+0x192b28
fffff802`51b194b0 fffff802`51b19630     : ffff980c`cd530000 ffffc7ad`1671fa95 ffff980c`cd530000 fffff802`490b4180 : nvlddmkm+0xaf451
fffff802`51b194b8 ffff980c`cd530000     : ffffc7ad`1671fa95 ffff980c`cd530000 fffff802`490b4180 fffff802`6beabf05 : 0xfffff802`51b19630
fffff802`51b194c0 ffffc7ad`1671fa95     : ffff980c`cd530000 fffff802`490b4180 fffff802`6beabf05 ffff980c`cd430000 : 0xffff980c`cd530000
fffff802`51b194c8 ffff980c`cd530000     : fffff802`490b4180 fffff802`6beabf05 ffff980c`cd430000 ffff980c`cd530000 : 0xffffc7ad`1671fa95
fffff802`51b194d0 fffff802`490b4180     : fffff802`6beabf05 ffff980c`cd430000 ffff980c`cd530000 ffff980c`cd430000 : 0xffff980c`cd530000
fffff802`51b194d8 fffff802`6beabf05     : ffff980c`cd430000 ffff980c`cd530000 ffff980c`cd430000 ffff980c`cd530000 : 0xfffff802`490b4180
fffff802`51b194e0 ffff980c`cd430000     : ffff980c`cd530000 ffff980c`cd430000 ffff980c`cd530000 00000000`00000004 : nvlddmkm+0x9bf05
fffff802`51b194e8 ffff980c`cd530000     : ffff980c`cd430000 ffff980c`cd530000 00000000`00000004 00000000`000100ee : 0xffff980c`cd430000
fffff802`51b194f0 ffff980c`cd430000     : ffff980c`cd530000 00000000`00000004 00000000`000100ee 000000ff`00000100 : 0xffff980c`cd530000
fffff802`51b194f8 ffff980c`cd530000     : 00000000`00000004 00000000`000100ee 000000ff`00000100 00000000`00000001 : 0xffff980c`cd430000
fffff802`51b19500 00000000`00000004     : 00000000`000100ee 000000ff`00000100 00000000`00000001 fffff802`4c927a00 : 0xffff980c`cd530000
fffff802`51b19508 00000000`000100ee     : 000000ff`00000100 00000000`00000001 fffff802`4c927a00 fffff802`4bf8dea9 : 0x4
fffff802`51b19510 000000ff`00000100     : 00000000`00000001 fffff802`4c927a00 fffff802`4bf8dea9 fffff802`4c927a00 : 0x100ee
fffff802`51b19518 00000000`00000001     : fffff802`4c927a00 fffff802`4bf8dea9 fffff802`4c927a00 00000000`0c5792a5 : 0x000000ff`00000100
fffff802`51b19520 fffff802`4c927a00     : fffff802`4bf8dea9 fffff802`4c927a00 00000000`0c5792a5 00000096`5b8fdfd0 : 0x1
fffff802`51b19528 fffff802`4bf8dea9     : fffff802`4c927a00 00000000`0c5792a5 00000096`5b8fdfd0 00000096`795d44d0 : nt!KiInitialThread
fffff802`51b19530 00000096`d2c573d0     : 00000000`00000000 00000000`00000000 00000000`00000000 00000022`00000000 : nt!HvlGetReferenceTime+0x21
fffff802`51b19560 00000000`00000000     : 00000000`00000000 00000000`00000000 00000022`00000000 00000000`00000000 : 0x00000096`d2c573d0


SYMBOL_NAME:  nvlddmkm+192b28

MODULE_NAME: nvlddmkm

IMAGE_NAME:  nvlddmkm.sys

STACK_COMMAND:  .thread ; .cxr ; kb

BUCKET_ID_FUNC_OFFSET:  192b28

FAILURE_BUCKET_ID:  0x133_ISR_nvlddmkm!unknown_function

OS_VERSION:  10.0.19041.1

BUILDLAB_STR:  vb_release

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

FAILURE_ID_HASH:  {f97493a5-ea2b-23ca-a808-8602773c2a86}

Followup:     MachineOwner Hi @Map1e0823 ,

If you are using TF2.10v then supported CUDA and cuDNN are 11.2 and 8.1 respectively. Please refer [source](https://www.tensorflow.org/install/source#gpu).

It seems your environment have higher versions and it may raise compatibility errors. Please downgrade them to suitable tested versions.

Thanks!
 Hi I just tried lowering the version of CUDA11.2.2 and cuDNN8.1.1 but the program still crashes. ;_; Anyone normally uses A4000 to train tf? ;_; Hi @Map1e0823 ,

Since the model is migrated from TF1.x which we are not supporting now. I would recommend to Debug the model code using this guide on [migration_debugging](https://www.tensorflow.org/guide/migrate/migration_debugging). ? Thanks! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62633">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62633">No</a>

### Issue type

Support

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

binary

### TensorFlow version

tf 2.10.1 git_version v2.10.0-76-gfdfc646704c

### Custom code

Yes

### OS platform and distribution

Windows 10 22H2

### Mobile device

_No response_

### Python version

3.9

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

CUDA 11.8.0, cuDNN8.9.6.50

### GPU model and memory

RTXA4000 16G Memory

### Current behavior?

I was trying to train a StegaStamp model, which is coded by TF1. I modified it into the code of TF2(with compat.v1, disable eager excution and disable v2 behavior etc.) and the code can run in CPU mode (tens of thousands of epochs). But when I try to use the GPU for training, the program will get stuck when running for a few hundred or less than 2,000 epochs, and Windows will restart with a blue screen after dozens of seconds.
I also tried  downgrade the version of tf cuda cudnn etc. like tf2.7.0 cuda11.6 cudnn8.3.2(Available combinations given on the web). but the program just exits with "Process finished with exit code -1073740791 (0xC0000409)" and didn't raise any exception.

### Standalone code to reproduce the issue

```shell
StegaStamp on Github
```


### Relevant log output

WARNING:tensorflow:From D:\work\project\StegaStamp\venv\lib\site-packages\tensorflow\python\compat\v2_compat.py:107: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
2023-12-14 13:26:10.184254: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-12-14 13:26:11.931143: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 12588 MB memory:  -> device: 0, name: NVIDIA RTX A4000, pci bus id: 0000:01:00.0, compute capability: 8.6
WARNING:tensorflow:From D:\work\project\StegaStamp\venv\lib\site-packages\tensorflow\python\util\dispatch.py:1176: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
WARNING:tensorflow:From D:\work\project\StegaStamp\venv\lib\site-packages\tensorflow\python\util\dispatch.py:1176: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
2023-12-14 13:26:13.575786: W tensorflow/core/common_runtime/graph_constructor.cc:1526] Importing a graph with a lower producer version 27 into an existing graph with producer version 1205. Shape inference will have run different parts of the graph with different producer versions.
WARNING:tensorflow:From D:\work\project\StegaStamp\venv\lib\site-packages\tensorflow\python\training\rmsprop.py:188: calling Ones.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
2023-12-14 13:26:16.487836: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled
2023-12-14 13:26:19.443420: I tensorflow/core/util/cuda_solvers.cc:179] Creating GpuSolver handles for stream 0000020FC8436850
2023-12-14 13:26:21.158282: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
2023-12-14 13:26:22.260385: I tensorflow/stream_executor/cuda/cuda_dnn.cc:384] Loaded cuDNN version 8906