Process aborted when running `tf.gather` and `tf.compat.v1.gather` on GPU with large parameters and indices
Hi @oawxkw ,

I have tested the code with both CPU and GPU runtimes. On CPU it is raising exception and on GPU its giving OOM error. I have executed the GPU code multiple runs and its OOM error only due to large input size. 

Please refer to attached [cpu-gist](https://colab.sandbox.google.com/gist/SuryanarayanaY/b94647d4d7875c851896a82436d2fd4d/61780_cpu.ipynb) and [gpu-gist](https://colab.sandbox.google.com/gist/SuryanarayanaY/f0d6e2aa0c532b01d5e2510bbb587cfd/61780_gpu.ipynb) here.

I am unable to replicate GPU behaviour reported by you. Could you please cross check and confirm the GPU behaviour. If OOM error occurs then it should fine and API working as expected.

Thanks! 
I think this is probably a GPU/memory specific issue, and I'm using a 24GiB RTX 3090 GPU with the latest `tensorflow/tensorflow:2.14.0rc1-gpu` docker image.
I've tried to run the code with different parameters (only the last dimension is changed).
The Check failed error only occurs when the last dimension of `params` is 10 or 14, while the OOM error occurs when the last dimension is 36.
Here are the outputs of different parameters:

```python
params = tf.saturate_cast(tf.random.uniform([13, 15, 7, 13, 8], minval=-1024, maxval=1024, dtype=tf.int64), dtype=tf.half)

# output:
# v2.14.0-rc0-34-gdd01672d9a9 2.14.0-rc1
# [PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'), PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
# Success!
```

```python
params = tf.saturate_cast(tf.random.uniform([13, 15, 7, 13, 10], minval=-1024, maxval=1024, dtype=tf.int64), dtype=tf.half)

# output:
# v2.14.0-rc0-34-gdd01672d9a9 2.14.0-rc1
# [PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'), PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
# 2023-09-04 08:27:16.315778: F ./tensorflow/core/util/gpu_launch_config.h:160] Check failed: work_element_count >= 0 (0 vs. -1619294296)
# Aborted (core dumped)
```

```python
params = tf.saturate_cast(tf.random.uniform([13, 15, 7, 13, 14], minval=-1024, maxval=1024, dtype=tf.int64), dtype=tf.half)

# output:
# v2.14.0-rc0-34-gdd01672d9a9 2.14.0-rc1
# [PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'), PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
# 2023-09-04 08:18:55.709111: F ./tensorflow/core/util/gpu_launch_config.h:160] Check failed: work_element_count >= 0 (0 vs. -549025096)
# Aborted (core dumped)
```

```python
params = tf.saturate_cast(tf.random.uniform([13, 15, 7, 13, 18], minval=-1024, maxval=1024, dtype=tf.int64), dtype=tf.half)

# output:
# v2.14.0-rc0-34-gdd01672d9a9 2.14.0-rc1
# [PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'), PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
# Success!
```

```python
params = tf.saturate_cast(tf.random.uniform([13, 15, 7, 13, 36], minval=-1024, maxval=1024, dtype=tf.int64), dtype=tf.half)

# output:
# v2.14.0-rc0-34-gdd01672d9a9 2.14.0-rc1
# [PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'), PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
# Error: {{function_node __wrapped__GatherV2_device_/job:localhost/replica:0/task:0/device:GPU:0}} OOM when allocating tensor with shape[11,12,6,15,11,3,15,7,13,36] and type half on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:GatherV2] name: 
# Success!
```
 Hello @SuryanarayanaY ,

I finally observed the same behavior on colab with the following code snippet, and you can reproduce it by running the code on colab with GPU enabled.
The Check failed error only occurs when the last dimension of `params` is 5 (in T4 GPU), while the OOM error occurs when the last dimension is greater than 6.

```python
import tensorflow as tf
print(tf.version.GIT_VERSION, tf.version.VERSION, flush=True)

try:
    validate_indices = False # True
    params = tf.saturate_cast(tf.random.uniform([13, 15, 7, 13, 5], minval=-1024, maxval=1024, dtype=tf.int64), dtype=tf.half)
    indices = tf.saturate_cast(tf.random.uniform([11, 12, 6, 15, 11, 3], minval=-1024, maxval=1024, dtype=tf.int64), dtype=tf.int64)
    res = tf.gather(
        validate_indices=validate_indices,
        params=params,
        indices=indices,
    )
except Exception as e:
    print("Error:", str(e), flush=True)
print("Success!", flush=True)
```
![image](https://github.com/tensorflow/tensorflow/assets/48113733/12030143-2a28-4abf-8a10-cf45bcf9e1ea)
 @oawxkw ,

Thanks for your time in submitting the check-fail case. I have replicated the reported behaviour now. As you said when the last dimension of params i.e  when `params.shape[-1] = 5` then only check fail followed by crash is happening. Attached screenshot below.It needs to be checked. Also security issues has to be reported as mentioned in [SECURITY.md](https://github.com/tensorflow/tensorflow/blob/master/SECURITY.md) .

<img width="1498" alt="Screenshot 2023-09-07 at 5 40 18 PM" src="https://github.com/tensorflow/tensorflow/assets/116063290/0ac4af2d-c8c4-4178-bff6-bc0d6cf7aff7">


For other values `params.shape[-1] < 5` or `params.shape[-1] > 5`, the API working as intended. Attached [gist](https://colab.sandbox.google.com/gist/SuryanarayanaY/07e18cedb617437010055b49a33e2d76/61780_gpu_r1.ipynb) for reference.
### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

binary

### TensorFlow version

2.14.0-rc1

### Custom code

Yes

### OS platform and distribution

_No response_

### Mobile device

_No response_

### Python version

_No response_

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

Process aborted when running `tf.gather` and `tf.compat.v1.gather` on GPU with certain parameters and indices, while it throws an Exception on CPU.
It always happens on GPU favor when the `params` and `indices` are large, regardless of the `validate_indices` value.
Besides, the Check failed error still occurs when `validate_indices` is set to `True`, and it should be an Exception instead.
I think the root cause of this behavior should be related to the bug in #60276 and #60756, which face a Check failed error in `tf.raw_ops.Gather/V2` op.


### Standalone code to reproduce the issue

```python
import tensorflow as tf
print(tf.version.GIT_VERSION, tf.version.VERSION, flush=True)

try:
    validate_indices = False # True
    params = tf.saturate_cast(tf.random.uniform([13, 15, 7, 13, 14], minval=-1024, maxval=1024, dtype=tf.int64), dtype=tf.half)
    indices = tf.saturate_cast(tf.random.uniform([11, 12, 6, 15, 11, 3], minval=-1024, maxval=1024, dtype=tf.int64), dtype=tf.int64)
    res = tf.gather(
        validate_indices=validate_indices,
        params=params,
        indices=indices,
    )
    # res = tf.compat.v1.gather(
    #     validate_indices=validate_indices,
    #     params=params,
    #     indices=indices,
    # )
except Exception as e:
    print("Error:", str(e), flush=True)
print("Success!", flush=True)
```


### Relevant log output

On GPU, it throws a Check failed and core dumped.

```text
v2.14.0-rc0-34-gdd01672d9a9 2.14.0-rc1
2023-09-03 13:08:24.487989: F ./tensorflow/core/util/gpu_launch_config.h:160] Check failed: work_element_count >= 0 (0 vs. -549025096)
Aborted (core dumped)
```

On CPU, it throws a Exception.

```text
v2.14.0-rc0-34-gdd01672d9a9 2.14.0-rc1
Error: {{function_node __wrapped__GatherV2_device_/job:localhost/replica:0/task:0/device:CPU:0}} indices[7,11,2,8,5,1] = 424 is not in [0, 13) [Op:GatherV2] name: 
Success!
```
