A check fail can be triggered in Conv3DBackpropFilterV2
Hi @shijy16 Thank you for reporting this issue!

I'm facing a different error when tried to replicate the issue in Ubuntu 20.04 using tf-nightly(2.13.0-dev20230402).  Please find the output log below.
![image](https://user-images.githubusercontent.com/98147397/229462081-cc56c52f-3969-4bef-99c3-208370cc90fd.png)
The issue has been resolved in the lastest tf-nightly version(2.13.0-dev20230402). Thank you! @synandi 
Thanks for your response!
I tried to replicate the issue again on `2.13.0-dev20230402` and can still replicate the crash. 
And it seems interesting because sometimes an `InvalidArgumentError` was thrown, and sometimes it ended with a crash due to check-fail. This is the output log:
![1680517518698](https://user-images.githubusercontent.com/31403391/229483569-6d5b1d92-c36b-4115-8eba-169584a30349.jpg)
This behaviour seems to be caused by using uninitailized values (just a guess).
Please executed the code for more times or simply add a for loop to replicate the crash as following:
````python
import tensorflow as tf
with tf.device("CPU"):
    for i in range(100):
        strides = [1, 1, 1, 1, 1]
        padding = "VALID"
        data_format = "NCDHW"
        dilations = [1, 1, 1, 1, 1]
        input = tf.random.uniform([1], dtype=tf.bfloat16, minval=-1024, maxval=1024)
        filter_sizes = tf.saturate_cast(tf.random.uniform([1], minval=-1024, maxval=1024, dtype=tf.int64), dtype=tf.int32)
        out_backprop = tf.random.uniform([2, 12, 3, 5, 10], dtype=tf.bfloat16, minval=-1024, maxval=1024)
        try:
            res = tf.raw_ops.Conv3DBackpropFilterV2(
                strides=strides,
                padding=padding,
                data_format=data_format,
                dilations=dilations,
                input=input,
                filter_sizes=filter_sizes,
                out_backprop=out_backprop,
            )
        except:
            pass
```` @shijy16, I tried  executing the code multiple times using tf-nightly(2.13.0-dev20230402). I'm facing `InvalidArgumentError` without crash. Please check the log below 

```
(tf) ynandi@sindhu-ubuntu-2005:~$ python 60195.py
2023-04-04 09:40:05.654152: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-04-04 09:40:06.482887: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-04-04 09:40:09.522641: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13492 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5
2023-04-04 09:40:09.524565: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 13599 MB memory:  -> device: 1, name: Tesla T4, pci bus id: 0000:00:05.0, compute capability: 7.5
Traceback (most recent call last):
  File "/home/ynandi/60195.py", line 13, in <module>
    res = tf.raw_ops.Conv3DBackpropFilterV2(
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/util/tf_export.py", line 413, in wrapper
    return f(**kwargs)
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 2209, in conv3d_backprop_filter_v2
    _ops.raise_from_not_ok_status(e, name)
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status
    raise core._status_to_exception(e) from None  # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__Conv3DBackpropFilterV2_device_/job:localhost/replica:0/task:0/device:CPU:0}} Conv3DBackpropFilterOpV2 only supports NDHWC on the CPU. [Op:Conv3DBackpropFilterV2] name: 
(tf) ynandi@sindhu-ubuntu-2005:~$ python 60195.py
2023-04-04 09:40:14.139250: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-04-04 09:40:14.970103: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-04-04 09:40:18.011821: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13492 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5
2023-04-04 09:40:18.013777: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 13599 MB memory:  -> device: 1, name: Tesla T4, pci bus id: 0000:00:05.0, compute capability: 7.5
Traceback (most recent call last):
  File "/home/ynandi/60195.py", line 13, in <module>
    res = tf.raw_ops.Conv3DBackpropFilterV2(
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/util/tf_export.py", line 413, in wrapper
    return f(**kwargs)
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 2209, in conv3d_backprop_filter_v2
    _ops.raise_from_not_ok_status(e, name)
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status
    raise core._status_to_exception(e) from None  # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__Conv3DBackpropFilterV2_device_/job:localhost/replica:0/task:0/device:CPU:0}} Conv3DBackpropFilterOpV2 only supports NDHWC on the CPU. [Op:Conv3DBackpropFilterV2] name: 
(tf) ynandi@sindhu-ubuntu-2005:~$ python 60195.py
2023-04-04 09:40:22.329471: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-04-04 09:40:23.166682: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-04-04 09:40:26.442930: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13492 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5
2023-04-04 09:40:26.444873: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 13599 MB memory:  -> device: 1, name: Tesla T4, pci bus id: 0000:00:05.0, compute capability: 7.5
Traceback (most recent call last):
  File "/home/ynandi/60195.py", line 13, in <module>
    res = tf.raw_ops.Conv3DBackpropFilterV2(
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/util/tf_export.py", line 413, in wrapper
    return f(**kwargs)
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 2209, in conv3d_backprop_filter_v2
    _ops.raise_from_not_ok_status(e, name)
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status
    raise core._status_to_exception(e) from None  # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__Conv3DBackpropFilterV2_device_/job:localhost/replica:0/task:0/device:CPU:0}} Conv3DBackpropFilterOpV2 only supports NDHWC on the CPU. [Op:Conv3DBackpropFilterV2] name: 
(tf) ynandi@sindhu-ubuntu-2005:~$ python 60195.py
2023-04-04 09:40:30.577044: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-04-04 09:40:31.387829: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-04-04 09:40:34.419442: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13492 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5
2023-04-04 09:40:34.421327: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 13599 MB memory:  -> device: 1, name: Tesla T4, pci bus id: 0000:00:05.0, compute capability: 7.5
Traceback (most recent call last):
  File "/home/ynandi/60195.py", line 13, in <module>
    res = tf.raw_ops.Conv3DBackpropFilterV2(
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/util/tf_export.py", line 413, in wrapper
    return f(**kwargs)
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 2209, in conv3d_backprop_filter_v2
    _ops.raise_from_not_ok_status(e, name)
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status
    raise core._status_to_exception(e) from None  # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__Conv3DBackpropFilterV2_device_/job:localhost/replica:0/task:0/device:CPU:0}} Conv3DBackpropFilterOpV2 only supports NDHWC on the CPU. [Op:Conv3DBackpropFilterV2] name: 
(tf) ynandi@sindhu-ubuntu-2005:~$ python 60195.py
2023-04-04 09:40:37.197768: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-04-04 09:40:38.015651: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-04-04 09:40:41.037032: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13492 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5
2023-04-04 09:40:41.039006: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 13599 MB memory:  -> device: 1, name: Tesla T4, pci bus id: 0000:00:05.0, compute capability: 7.5
Traceback (most recent call last):
  File "/home/ynandi/60195.py", line 13, in <module>
    res = tf.raw_ops.Conv3DBackpropFilterV2(
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/util/tf_export.py", line 413, in wrapper
    return f(**kwargs)
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 2209, in conv3d_backprop_filter_v2
    _ops.raise_from_not_ok_status(e, name)
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status
    raise core._status_to_exception(e) from None  # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__Conv3DBackpropFilterV2_device_/job:localhost/replica:0/task:0/device:CPU:0}} Conv3DBackpropFilterOpV2 only supports NDHWC on the CPU. [Op:Conv3DBackpropFilterV2] name: 
(tf) ynandi@sindhu-ubuntu-2005:~$ python 60195.py
2023-04-04 09:40:43.719981: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-04-04 09:40:44.536964: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-04-04 09:40:47.558622: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13492 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5
2023-04-04 09:40:47.560566: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 13599 MB memory:  -> device: 1, name: Tesla T4, pci bus id: 0000:00:05.0, compute capability: 7.5
Traceback (most recent call last):
  File "/home/ynandi/60195.py", line 13, in <module>
    res = tf.raw_ops.Conv3DBackpropFilterV2(
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/util/tf_export.py", line 413, in wrapper
    return f(**kwargs)
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 2209, in conv3d_backprop_filter_v2
    _ops.raise_from_not_ok_status(e, name)
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status
    raise core._status_to_exception(e) from None  # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__Conv3DBackpropFilterV2_device_/job:localhost/replica:0/task:0/device:CPU:0}} Conv3DBackpropFilterOpV2 only supports NDHWC on the CPU. [Op:Conv3DBackpropFilterV2] name: 
(tf) ynandi@sindhu-ubuntu-2005:~$ python 60195.py
2023-04-04 09:40:50.022309: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-04-04 09:40:50.838318: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-04-04 09:40:53.889195: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13492 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5
2023-04-04 09:40:53.891310: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 13599 MB memory:  -> device: 1, name: Tesla T4, pci bus id: 0000:00:05.0, compute capability: 7.5
Traceback (most recent call last):
  File "/home/ynandi/60195.py", line 13, in <module>
    res = tf.raw_ops.Conv3DBackpropFilterV2(
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/util/tf_export.py", line 413, in wrapper
    return f(**kwargs)
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 2209, in conv3d_backprop_filter_v2
    _ops.raise_from_not_ok_status(e, name)
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status
    raise core._status_to_exception(e) from None  # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__Conv3DBackpropFilterV2_device_/job:localhost/replica:0/task:0/device:CPU:0}} Conv3DBackpropFilterOpV2 only supports NDHWC on the CPU. [Op:Conv3DBackpropFilterV2] name: 
(tf) ynandi@sindhu-ubuntu-2005:~$ python 60195.py
2023-04-04 09:40:57.744063: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-04-04 09:40:58.559910: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-04-04 09:41:01.646660: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13498 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5
2023-04-04 09:41:01.648587: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 13599 MB memory:  -> device: 1, name: Tesla T4, pci bus id: 0000:00:05.0, compute capability: 7.5
Traceback (most recent call last):
  File "/home/ynandi/60195.py", line 13, in <module>
    res = tf.raw_ops.Conv3DBackpropFilterV2(
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/util/tf_export.py", line 413, in wrapper
    return f(**kwargs)
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 2209, in conv3d_backprop_filter_v2
    _ops.raise_from_not_ok_status(e, name)
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status
    raise core._status_to_exception(e) from None  # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__Conv3DBackpropFilterV2_device_/job:localhost/replica:0/task:0/device:CPU:0}} Conv3DBackpropFilterOpV2 only supports NDHWC on the CPU. [Op:Conv3DBackpropFilterV2] name: 
(tf) ynandi@sindhu-ubuntu-2005:~$ python 60195.py
2023-04-04 09:41:04.205750: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-04-04 09:41:05.019126: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-04-04 09:41:08.034700: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13498 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5
2023-04-04 09:41:08.036647: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 13599 MB memory:  -> device: 1, name: Tesla T4, pci bus id: 0000:00:05.0, compute capability: 7.5
Traceback (most recent call last):
  File "/home/ynandi/60195.py", line 13, in <module>
    res = tf.raw_ops.Conv3DBackpropFilterV2(
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/util/tf_export.py", line 413, in wrapper
    return f(**kwargs)
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 2209, in conv3d_backprop_filter_v2
    _ops.raise_from_not_ok_status(e, name)
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status
    raise core._status_to_exception(e) from None  # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__Conv3DBackpropFilterV2_device_/job:localhost/replica:0/task:0/device:CPU:0}} Conv3DBackpropFilterOpV2 only supports NDHWC on the CPU. [Op:Conv3DBackpropFilterV2] name: 
(tf) ynandi@sindhu-ubuntu-2005:~$ python 60195.py
2023-04-04 09:41:10.367393: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-04-04 09:41:11.197626: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-04-04 09:41:14.278666: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13498 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5
2023-04-04 09:41:14.280686: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 13599 MB memory:  -> device: 1, name: Tesla T4, pci bus id: 0000:00:05.0, compute capability: 7.5
Traceback (most recent call last):
  File "/home/ynandi/60195.py", line 13, in <module>
    res = tf.raw_ops.Conv3DBackpropFilterV2(
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/util/tf_export.py", line 413, in wrapper
    return f(**kwargs)
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 2209, in conv3d_backprop_filter_v2
    _ops.raise_from_not_ok_status(e, name)
  File "/home/ynandi/miniconda3/envs/tf/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status
    raise core._status_to_exception(e) from None  # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__Conv3DBackpropFilterV2_device_/job:localhost/replica:0/task:0/device:CPU:0}} Conv3DBackpropFilterOpV2 only supports NDHWC on the CPU. [Op:Conv3DBackpropFilterV2] name: 
(tf) ynandi@sindhu-ubuntu-2005:~$ 
```

Thank you! @synandi 
Hi, sorry for that.

I just noticed that the bug can only be replicated when Intel OneDNN is enabled. In the following code, Intel OneDNN is enabled, it ended with a crash.
![image](https://user-images.githubusercontent.com/31403391/229795882-de150cc6-39ca-466f-8c77-79a58ba0c8dc.png)
In the following code, Intel OneDNN is disabled by setting environment viriable  `TF_ENABLE_ONEDNN_OPTS` as 0, and it ended normally.
![image](https://user-images.githubusercontent.com/31403391/229797395-640e86c3-fb5a-422d-91f0-c21d89b50a61.png)
Since your TensorFlow binary seems to be compiled without Intel OneDNN support (there is no output log as below when tensorflow is loaded), the bug cannot be replicated.
````
oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
```` @shijy16 Please don't file vulnerabilities on GitHub. Please consult https://github.com/tensorflow/tensorflow/blob/master/SECURITY.md and follow rules for responsible disclosure. @mihaimaruseac 
Hi, the bugs I reported here are mostly `check fails`, which are not considered as vulnerabilities according to the rules below. Thus I think reporting them here is okay.
````
If an assertion failure only leads to program termination and no other exploits, we will no longer consider assertion failures (e.g., CHECK-fails) as vulnerabilities. 
```` Hi @shijy16 ,

The issue has been resolved already. Please check the attached [gist](https://colab.sandbox.google.com/gist/SuryanarayanaY/e536570d5a78a1c9a8facedc7f49aeb0/60195_nightly_success.ipynb) with Tf2.16. This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60195">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60195">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

Yes

### Source

binary

### Tensorflow Version

2.13.0-dev20230331

### Custom Code

Yes

### OS Platform and Distribution

Linux Ubuntu 20.04

### Mobile device

_No response_

### Python version

3.10

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

CUDA 11.5

### GPU model and memory

_No response_

### Current Behaviour?

```shell
The following code can trigger a crash in `tf.raw_ops.Conv3DBackpropFilterV2` due to check-fail in the latest version of TensorFlow.
```


### Standalone code to reproduce the issue

```shell
import tensorflow as tf
with tf.device("CPU"):
    strides = [1, 1, 1, 1, 1]
    padding = "VALID"
    data_format = "NCDHW"
    dilations = [1, 1, 1, 1, 1]
    input = tf.random.uniform([1], dtype=tf.bfloat16, minval=-1024, maxval=1024)
    filter_sizes = tf.saturate_cast(tf.random.uniform([1], minval=-1024, maxval=1024, dtype=tf.int64), dtype=tf.int32)
    out_backprop = tf.random.uniform([2, 12, 3, 5, 10], dtype=tf.bfloat16, minval=-1024, maxval=1024)
    res = tf.raw_ops.Conv3DBackpropFilterV2(
        strides=strides,
        padding=padding,
        data_format=data_format,
        dilations=dilations,
        input=input,
        filter_sizes=filter_sizes,
        out_backprop=out_backprop,
    )
```


### Relevant log output

```shell
2023-04-01 15:05:13.595051: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-04-01 15:05:13.597635: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-04-01 15:05:13.640719: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-04-01 15:05:13.641231: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-04-01 15:05:14.394924: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2.13.0-dev20230331
2023-04-01 15:05:15.220833: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1956] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2023-04-01 15:05:15.267117: F ./tensorflow/core/util/tensor_format.h:427] Check failed: index >= 0 && index < num_total_dims Invalid index from the dimension: 1, 1, C
Aborted (core dumped)
```
</details>