Check failure when running tf.compat.v1.layers.MaxPooling1D
Hi @dmc1778,

I have tested the code with tf version 2.12 and tf-nightly and its working fine by raising exception, please find the [gist](https://colab.sandbox.google.com/gist/Varsha-anjanappa/a9cbdf4d498596ff6c234d194ad7ae33/61642.ipynb) here. 

Thank you! > 

Thanks @Varsha-anjanappa. On my side (2.13.0):

```
2023-08-21 01:03:23.808421: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:432] Loaded cuDNN version 8600
2023-08-21 01:03:23.808473: F tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:983] Check failed: cudnnSetPoolingNdDescriptor( handle_.get(), (pooling_descriptor.mode() == dnn::PoolingMode::kMaximum ? cudnn_max_pooling_mode : CUDNN_POOLING_AVERAGE_COUNT_EXCLUDE_PADDING), propagate_nans ? CUDNN_PROPAGATE_NAN : CUDNN_NOT_PROPAGATE_NAN, nd, shape.data(), padding.data(), strides.data()) == CUDNN_STATUS_SUCCESS (3 vs. 0)
Aborted
``` Hi @dmc1778 ,

Tested the code on colab with tf version 2.13 as well, it is working fine. Please find the [gist](https://colab.sandbox.google.com/gist/Varsha-anjanappa/7eb2785fcf9750d1bb628a877be1f624/61642_v2.ipynb).

Thank you!!

 > 

My session on Colab is still crashing. Please find the attached snapshot:
![Screenshot from 2023-08-21 01-54-10](https://github.com/tensorflow/tensorflow/assets/17060581/17f02afd-c995-48c9-bf4e-df7970b9458d)

Log:

```
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.866 NotebookApp] Searching ['/root/.jupyter', '/root/.local/etc/jupyter', '/usr/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files","time":"2023-08-21T05:53:09.867Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.866 NotebookApp] Searching ['/root/.jupyter', '/root/.local/etc/jupyter', '/usr/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files","time":"2023-08-21T05:53:09.870Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.866 NotebookApp] Looking for jupyter_config in /etc/jupyter","time":"2023-08-21T05:53:09.868Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.878 NotebookApp] Looking for jupyter_config in /usr/local/etc/jupyter","time":"2023-08-21T05:53:09.882Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.883 NotebookApp] Looking for jupyter_config in /usr/etc/jupyter","time":"2023-08-21T05:53:09.887Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.883 NotebookApp] Looking for jupyter_config in /root/.local/etc/jupyter","time":"2023-08-21T05:53:09.887Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.884 NotebookApp] Looking for jupyter_config in /root/.jupyter","time":"2023-08-21T05:53:09.887Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.866 NotebookApp] Looking for jupyter_config in /etc/jupyter","time":"2023-08-21T05:53:09.871Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.869 NotebookApp] Looking for jupyter_config in /usr/local/etc/jupyter","time":"2023-08-21T05:53:09.873Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.870 NotebookApp] Looking for jupyter_config in /usr/etc/jupyter","time":"2023-08-21T05:53:09.882Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.870 NotebookApp] Looking for jupyter_config in /root/.local/etc/jupyter","time":"2023-08-21T05:53:09.882Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.871 NotebookApp] Looking for jupyter_config in /root/.jupyter","time":"2023-08-21T05:53:09.883Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.871 NotebookApp] Looking for jupyter_notebook_config in /etc/jupyter","time":"2023-08-21T05:53:09.883Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.875 NotebookApp] Loaded config file: /etc/jupyter/jupyter_notebook_config.py","time":"2023-08-21T05:53:09.884Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.876 NotebookApp] Looking for jupyter_notebook_config in /usr/local/etc/jupyter","time":"2023-08-21T05:53:09.886Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.876 NotebookApp] Loaded config file: /usr/local/etc/jupyter/jupyter_notebook_config.json","time":"2023-08-21T05:53:09.886Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.877 NotebookApp] Looking for jupyter_notebook_config in /usr/etc/jupyter","time":"2023-08-21T05:53:09.887Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.877 NotebookApp] Looking for jupyter_notebook_config in /root/.local/etc/jupyter","time":"2023-08-21T05:53:09.888Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.877 NotebookApp] Looking for jupyter_notebook_config in /root/.jupyter","time":"2023-08-21T05:53:09.891Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.879 NotebookApp] Loaded config file: /root/.jupyter/jupyter_notebook_config.py","time":"2023-08-21T05:53:09.891Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.889 NotebookApp] Looking for jupyter_notebook_config in /etc/jupyter","time":"2023-08-21T05:53:09.889Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.889 NotebookApp] Loaded config file: /etc/jupyter/jupyter_notebook_config.py","time":"2023-08-21T05:53:09.891Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.890 NotebookApp] Looking for jupyter_notebook_config in /usr/local/etc/jupyter","time":"2023-08-21T05:53:09.892Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.890 NotebookApp] Loaded config file: /usr/local/etc/jupyter/jupyter_notebook_config.json","time":"2023-08-21T05:53:09.892Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.890 NotebookApp] Looking for jupyter_notebook_config in /usr/etc/jupyter","time":"2023-08-21T05:53:09.892Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.890 NotebookApp] Looking for jupyter_notebook_config in /root/.local/etc/jupyter","time":"2023-08-21T05:53:09.892Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.890 NotebookApp] Looking for jupyter_notebook_config in /root/.jupyter","time":"2023-08-21T05:53:09.892Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 05:53:09.891 NotebookApp] Loaded config file: /root/.jupyter/jupyter_notebook_config.py","time":"2023-08-21T05:53:09.895Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"    \t/etc/jupyter/jupyter_notebook_config.json","time":"2023-08-21T05:53:10.735Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"    \t/usr/local/etc/jupyter/jupyter_notebook_config.d/panel-client-jupyter.json","time":"2023-08-21T05:53:10.745Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"    \t/usr/local/etc/jupyter/jupyter_notebook_config.json","time":"2023-08-21T05:53:10.745Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"    \t/usr/etc/jupyter/jupyter_notebook_config.json","time":"2023-08-21T05:53:10.746Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"    \t/root/.local/etc/jupyter/jupyter_notebook_config.json","time":"2023-08-21T05:53:10.747Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"    \t/root/.jupyter/jupyter_notebook_config.json","time":"2023-08-21T05:53:10.747Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret","time":"2023-08-21T05:53:10.765Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"Authentication of /metrics is OFF, since other authentication is disabled.","time":"2023-08-21T05:53:10.771Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"google.colab serverextension initialized.","time":"2023-08-21T05:53:10.836Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"    \t/etc/jupyter/jupyter_notebook_config.json","time":"2023-08-21T05:53:10.878Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"    \t/usr/local/etc/jupyter/jupyter_notebook_config.d/panel-client-jupyter.json","time":"2023-08-21T05:53:10.880Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"    \t/usr/local/etc/jupyter/jupyter_notebook_config.json","time":"2023-08-21T05:53:10.880Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"    \t/usr/etc/jupyter/jupyter_notebook_config.json","time":"2023-08-21T05:53:10.882Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"    \t/root/.local/etc/jupyter/jupyter_notebook_config.json","time":"2023-08-21T05:53:10.883Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"    \t/root/.jupyter/jupyter_notebook_config.json","time":"2023-08-21T05:53:10.884Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret","time":"2023-08-21T05:53:10.893Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"Authentication of /metrics is OFF, since other authentication is disabled.","time":"2023-08-21T05:53:10.894Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"google.colab serverextension initialized.","time":"2023-08-21T05:53:10.945Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"Serving notebooks from local directory: /","time":"2023-08-21T05:53:15.513Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"Jupyter Notebook 6.5.5 is running at:","time":"2023-08-21T05:53:15.514Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"Serving notebooks from local directory: /","time":"2023-08-21T05:53:15.521Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"Jupyter Notebook 6.5.5 is running at:","time":"2023-08-21T05:53:15.521Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"http://172.28.0.2:9000/","time":"2023-08-21T05:53:15.521Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).","time":"2023-08-21T05:53:15.522Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"http://172.28.0.12:9000/","time":"2023-08-21T05:53:15.523Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).","time":"2023-08-21T05:53:15.523Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"Kernel started: 7be3ebb5-8079-41dd-a058-72f1e271f55f, name: python3","time":"2023-08-21T05:53:32.402Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-21 05:53:38.426472: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.","time":"2023-08-21T05:53:38.426Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.","time":"2023-08-21T05:53:38.426Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-21 05:53:39.614903: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT","time":"2023-08-21T05:53:39.615Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-21 05:53:43.858531: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-21T05:53:43.858Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-21 05:53:44.410103: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-21T05:53:44.410Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-21 05:53:44.410567: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-21T05:53:44.410Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-21 05:53:44.413324: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-21T05:53:44.413Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-21 05:53:44.413957: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-21T05:53:44.414Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-21 05:53:44.414476: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-21T05:53:44.414Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-21 05:53:48.373843: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-21T05:53:48.374Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-21 05:53:48.380854: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-21T05:53:48.381Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-21 05:53:48.381791: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-21T05:53:48.381Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-21 05:53:48.382383: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:47] Overriding orig_value setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.","time":"2023-08-21T05:53:48.382Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-21 05:53:48.382464: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13664 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5","time":"2023-08-21T05:53:48.382Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-21 05:53:53.331521: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:424] Loaded cuDNN version 8900","time":"2023-08-21T05:53:53.331Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-21 05:53:53.332580: F tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:959] Check failed: cudnnSetPoolingNdDescriptor( handle_.get(), (pooling_descriptor.mode() == dnn::PoolingMode::kMaximum ? cudnn_max_pooling_mode : CUDNN_POOLING_AVERAGE_COUNT_EXCLUDE_PADDING), propagate_nans ? CUDNN_PROPAGATE_NAN : CUDNN_NOT_PROPAGATE_NAN, nd, shape.data(), padding.data(), strides.data()) == CUDNN_STATUS_SUCCESS (3 vs. 0)","time":"2023-08-21T05:53:53.332Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"KernelRestarter: restarting kernel (1/5), keep random ports","time":"2023-08-21T05:53:56.402Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"WARNING:root:kernel 7be3ebb5-8079-41dd-a058-72f1e271f55f restarted","time":"2023-08-21T05:53:56.402Z","v":0}
``` Hi @dmc1778 ,

As you can see the below snapshot, It is executing as expected, please refer to the gist given [here](https://github.com/tensorflow/tensorflow/issues/61642#issuecomment-1685691779).

![image](https://github.com/tensorflow/tensorflow/assets/137163810/7caad80a-20cf-41b2-ac84-cf66513f3234)

Thank you!!
 > Hi @dmc1778 ,
> 
> As you can see the below snapshot, It is executing as expected, please refer to the gist given [here](https://github.com/tensorflow/tensorflow/issues/61642#issuecomment-1685691779).
> 
> ![image](https://user-images.githubusercontent.com/137163810/262218562-7caad80a-20cf-41b2-ac84-cf66513f3234.png)
> 
> Thank you!!

This is a serious issue as we get different outputs.  Hi @dmc1778 ,

With CPU runtime the code executes fine and attached [gist](https://colab.sandbox.google.com/gist/SuryanarayanaY/267620ee9839af8f75f15797c51356a4/61642_cpu.ipynb) for reference.

With GPU runtime check fail followed by crash happening. Attached screenshot for below.

<img width="1506" alt="Screenshot 2023-08-23 at 4 46 03 PM" src="https://github.com/tensorflow/tensorflow/assets/116063290/6ff947dc-d562-44e9-b8bc-57e980fa6aef">


### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

binary

### TensorFlow version

2.13.0

### Custom code

Yes

### OS platform and distribution

_No response_

### Mobile device

_No response_

### Python version

_No response_

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

Due to feeding Large list elements

### Standalone code to reproduce the issue

```shell
import tensorflow as tf
import os
import numpy as np
try:
  pool_size_0 = 1e+38
  pool_size = [pool_size_0,]
  strides_0 = 2
  strides = [strides_0,]
  padding = "same"
  data_format = "channels_last"
  arg_class = tf.compat.v1.layers.MaxPooling1D(pool_size=pool_size,strides=strides,padding=padding,data_format=data_format,)
  arg_input_0_tensor = tf.random.uniform([1, 5, 4], dtype=tf.float32)
  arg_input_0 = tf.identity(arg_input_0_tensor)
  arg_input = [arg_input_0,]
  out = arg_class(*arg_input)
except Exception as e:
  print("Error:"+str(e))
```
```


### Relevant log output

```shell
023-08-18 22:04:07.309328: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-08-18 22:04:07.824681: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-08-18 22:04:08.293378: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-18 22:04:08.311735: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-18 22:04:08.311868: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-18 22:04:08.313249: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-18 22:04:08.313383: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-18 22:04:08.313504: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-18 22:04:08.379287: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-18 22:04:08.379413: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-18 22:04:08.379501: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-08-18 22:04:08.379578: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 4308 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1660 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5
2023-08-18 22:04:08.473502: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:432] Loaded cuDNN version 8600
2023-08-18 22:04:08.473555: F tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:983] Check failed: cudnnSetPoolingNdDescriptor( handle_.get(), (pooling_descriptor.mode() == dnn::PoolingMode::kMaximum ? cudnn_max_pooling_mode : CUDNN_POOLING_AVERAGE_COUNT_EXCLUDE_PADDING), propagate_nans ? CUDNN_PROPAGATE_NAN : CUDNN_NOT_PROPAGATE_NAN, nd, shape.data(), padding.data(), strides.data()) == CUDNN_STATUS_SUCCESS (3 vs. 0)
Aborted

```
```
