A CUDA_ERROR_MISALIGNED_ADDRESS/CUDA_ERROR_ILLEGAL_ADDRESS can be triggered in SparseSplit
Hi @shijy16 ,
Thanks for reporting the issue!
I was able to replicate the issue in Colab using TF v2.11 and tf-nightly(2.13.0-dev20230320). Please find the gists [here(2.11)](https://colab.research.google.com/gist/tiruk007/cb271c3ff011e74b136b1dd4903b15b7/untitled162.ipynb) and [here(tf-nightly)](https://colab.research.google.com/gist/tiruk007/ddeb0d8fb97c30e1349c451fb4ea854b/untitled162.ipynb). It seems like we have to dig more into this issue, we will update soon here. 

Thank you! @SuryanarayanaY 
I was able to reproduce the issue on a virtual machine using TF v2.12.  Please find the below screenshots for reference:
![2](https://user-images.githubusercontent.com/111861663/228052283-c1a1303a-46e9-46fc-bd72-b3881b8772f4.png)
![1](https://user-images.githubusercontent.com/111861663/228052295-7bceb6eb-5207-48d6-b50b-f52128ef2185.png)

Thank you !
 Hi @shijy16 ,

This seems to be a vulnerability. Request you to report the issues as per policy here.(https://github.com/tensorflow/tensorflow/blob/master/SECURITY.md#reporting-vulnerabilities).

Please also refer to this [comment](https://github.com/tensorflow/tensorflow/issues/60121#issuecomment-1485230826) by the Developer regarding credit.

Thank you!

 > Hi @shijy16 ,
> 
> This seems to be a vulnerability. Request you to report the issues as per policy here.(https://github.com/tensorflow/tensorflow/blob/master/SECURITY.md#reporting-vulnerabilities).
> 
> Please also refer to this [comment](https://github.com/tensorflow/tensorflow/issues/60121#issuecomment-1485230826) by the Developer regarding credit.
> 
> Thank you!

Hi, I am sorry that I did not realize this might be a vulnerability. I've reported the bug to OSS VRP project. 
Thank you! Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60036">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60036">No</a>
 Hi @shijy16 ,

Thanks again for reporting this. Now the issue is fixed with t`f-nightly(2.13.0-dev20230420)` and the code successfully raises intended error.Please refer below logs. Thanks!

```
(base) suryanarayanay@surya-ubuntu20:~$ python 60036_latest.py 
2023-04-20 15:11:25.430153: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-04-20 15:11:27.576142: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2.13.0-dev20230420
2023-04-20 15:11:32.854960: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-04-20 15:11:33.329714: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-04-20 15:11:33.332278: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-04-20 15:11:33.335989: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-04-20 15:11:33.338301: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-04-20 15:11:33.340511: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-04-20 15:11:34.904200: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-04-20 15:11:34.906890: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-04-20 15:11:34.909200: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-04-20 15:11:34.911420: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13623 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5
2023-04-20 15:11:35.501617: W tensorflow/core/framework/op_kernel.cc:1828] OP_REQUIRES failed at sparse_split_op.cc:109 : INVALID_ARGUMENT: Number of elements in indices (4) and values (16) do not match
2023-04-20 15:11:35.501695: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:GPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: Number of elements in indices (4) and values (16) do not match
         [[{{node SparseSplit}}]]
Traceback (most recent call last):
  File "/home/suryanarayanay/60036_latest.py", line 21, in <module>
    res = tf.raw_ops.SparseSplit(
  File "/home/suryanarayanay/miniconda3/lib/python3.10/site-packages/tensorflow/python/util/tf_export.py", line 413, in wrapper
    return f(**kwargs)
  File "/home/suryanarayanay/miniconda3/lib/python3.10/site-packages/tensorflow/python/ops/gen_sparse_ops.py", line 2894, in sparse_split
    return sparse_split_eager_fallback(
  File "/home/suryanarayanay/miniconda3/lib/python3.10/site-packages/tensorflow/python/ops/gen_sparse_ops.py", line 2928, in sparse_split_eager_fallback
    _result = _execute.execute(b"SparseSplit", num_split + num_split +
  File "/home/suryanarayanay/miniconda3/lib/python3.10/site-packages/tensorflow/python/eager/execute.py", line 53, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__SparseSplit_num_split_30_device_/job:localhost/replica:0/task:0/device:GPU:0}} Number of elements in indices (4) and values (16) do not match
         [[{{node SparseSplit}}]] [Op:SparseSplit]
(base) suryanarayanay@surya-ubuntu20:~$ 
```
<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

Yes

### Source

binary

### Tensorflow Version

tf 2.13.0-dev20230317

### Custom Code

No

### OS Platform and Distribution

_No response_

### Mobile device

_No response_

### Python version

3.10

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

CUDA 11.5

### GPU model and memory

_No response_

### Current Behaviour?

```shell
A cuda memory corruption can be triggered by the following code in `tf.raw_ops.SparseSplit`. When the code is executing, a CUDA Memory Error would be reported, the gpu memory would be occupied, and it will never stop.
```


### Standalone code to reproduce the issue

```shell
import tensorflow as tf
for i in range(100):
    with tf.device("GPU:0"):
        num_split = 30
        split_dim = -1
        indices = tf.saturate_cast(tf.random.uniform([4, 8], minval=-1024, maxval=1024, dtype=tf.int64), dtype=tf.int64)
        values = tf.complex(tf.random.uniform([16], dtype=tf.float32, minval=-1024, maxval=1024),tf.random.uniform([16], dtype=tf.float32, minval=-1024, maxval=1024))
        shape = [108, 28, 108]
        res = tf.raw_ops.SparseSplit(
            num_split=num_split,
            split_dim=split_dim,
            indices=indices,
            values=values,
            shape=shape,
        )
```


### Relevant log output

```shell
2023-03-18 18:36:32.613986: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-03-18 18:36:32.661775: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-03-18 18:36:33.415445: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-03-18 18:36:35.026862: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 14577 MB memory:  -> device: 0, name: Tesla V100-PCIE-16GB, pci bus id: 0000:2f:00.0, compute capability: 7.0
2023-03-18 18:36:35.342047: E tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:789] failed to record completion event; therefore, failed to create inter-stream dependency
2023-03-18 18:36:35.342151: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:1160] failed to enqueue async memcpy from device to host: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered; host dst: 0x7f6d47200000; GPU src: 0x7f69b8000500; size: 8=0x82023-03-18 18:36:35.342182: E tensorflow/compiler/xla/stream_executor/stream.cc:336] Error recording event in stream: Error recording CUDA event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered; not marking stream as bad, as the Event object may be at fault. Monitor for further errors.
2023-03-18 18:36:35.342209: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:1033] could not wait stream on event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2023-03-18 18:36:35.342229: E tensorflow/compiler/xla/stream_executor/stream.cc:1120] Error waiting for event in stream: error recording waiting for CUDA event on stream 0x5562acfa15e0; not marking stream as bad, as the Event object may be at fault. Monitor for further errors.
2023-03-18 18:36:35.342248: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:615] unable to add host callback: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2023-03-18 18:36:35.342299: I tensorflow/compiler/xla/stream_executor/stream.cc:1108] [stream=0x556292e3cf30,impl=0x5562acb89830] did not wait for [stream=0x556292e3f420,impl=0x5562acb898d0]
2023-03-18 18:36:35.342323: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:1160] failed to enqueue async memcpy from device to host: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered; host dst: 0x7f6d47200100; GPU src: 0x7f69b8000600; size: 8=0x82023-03-18 18:36:35.342345: E tensorflow/compiler/xla/stream_executor/stream.cc:336] Error recording event in stream: Error recording CUDA event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered; not marking stream as bad, as the Event object may be at fault. Monitor for further errors.
2023-03-18 18:36:35.342367: I tensorflow/compiler/xla/stream_executor/stream.cc:1126] [stream=0x5562acfa15e0,impl=0x5562924676a0] did not wait for an event.
2023-03-18 18:36:35.342386: I tensorflow/compiler/xla/stream_executor/stream.cc:2393] [stream=0x5562acfa15e0,impl=0x5562924676a0] was in error state before adding host callback
2023-03-18 18:36:35.342403: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:615] unable to add host callback: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
```
</details>