tf.nn.embedding_lookup works fine in CPU mode, but lacks constraint checking in GPU mode
Hi @EgodPrime ,

This is due to the `tf.gather` Op, which is called internally by `embedding_lookup`. `tf.gather` Op never validate indices on GPU and on CPU indices always be validated.Hence on GPU there is no exception as raised by CPU. This is known issue and also documented in the tf.gather doc [here](https://www.tensorflow.org/api_docs/python/tf/gather#:~:text=The%20validate_indices%20argument%20has%20no%20effect.%20Indices%20are%20always%20validated%20on%20CPU%20and%20never%20validated%20on%20GPU.).

Please refer to the argument description of validate_indices under args section of [tf.gather](https://www.tensorflow.org/api_docs/python/tf/gather#:~:text=Caution%3A%20On%20CPU%2C%20if%20an%20out%20of%20bound%20index%20is%20found%2C%20an%20error%20is%20raised.%20On%20GPU%2C%20if%20an%20out%20of%20bound%20index%20is%20found%2C%20a%200%20is%20stored%20in%20the%20corresponding%20output%20value.) which is below.

> Caution: On CPU, if an out of bound index is found, an error is raised. On GPU, if an out of bound index is found, a 0 is stored in the corresponding output value.

Please also refer Developer [comment](https://github.com/tensorflow/tensorflow/issues/59724#issuecomment-1466806376) on same. > Hi @EgodPrime ,
> 
> This is due to the `tf.gather` Op, which is called internally by `embedding_lookup`. `tf.gather` Op never validate indices on GPU and on CPU indices always be validated.Hence on GPU there is no exception as raised by CPU. This is known issue and also documented in the tf.gather doc [here](https://www.tensorflow.org/api_docs/python/tf/gather#:~:text=The%20validate_indices%20argument%20has%20no%20effect.%20Indices%20are%20always%20validated%20on%20CPU%20and%20never%20validated%20on%20GPU.).
> 
> Please refer to the argument description of validate_indices under args section of [tf.gather](https://www.tensorflow.org/api_docs/python/tf/gather#:~:text=Caution%3A%20On%20CPU%2C%20if%20an%20out%20of%20bound%20index%20is%20found%2C%20an%20error%20is%20raised.%20On%20GPU%2C%20if%20an%20out%20of%20bound%20index%20is%20found%2C%20a%200%20is%20stored%20in%20the%20corresponding%20output%20value.) which is below.
> 
> > Caution: On CPU, if an out of bound index is found, an error is raised. On GPU, if an out of bound index is found, a 0 is stored in the corresponding output value.
> 
> Please also refer Developer [comment](https://github.com/tensorflow/tensorflow/issues/59724#issuecomment-1466806376) on same.

Thanks for your response. I appreciate the points you've mentioned. However, it would be beneficial if you could directly include some explanations about the constraints related to tf.nn.embedding_lookup to enhance clarity :) This is true for most ops that use indicies - it's too expensive to do constraint checking over the input on GPU, so we always just ignore bad indices.  It's still an invalid model that will result in undefined behavior. @EgodPrime,
The related [PR](https://github.com/tensorflow/tensorflow/pull/62734) which was raised for adding the caution not for the current behaviour has been merged and also please take a look at the above comment from the Developer for the same.

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/embedding_ops.py#L310

https://github.com/tensorflow/tensorflow/pull/62734
Thank you!
### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

binary

### TensorFlow version

tf2.14.0

### Custom code

Yes

### OS platform and distribution

Linux Ubuntu 20.04

### Mobile device

_No response_

### Python version

3.9

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

11.8

### GPU model and memory

_No response_

### Current behavior?

When calling tf.nn.embedding_lookup with an input that does not meet the constraints, the input error will be correctly determined in CPU mode, but the calculation will be completed normally in GPU mode and a result of all 0s will be obtained.

```python
import os

os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'

os.environ['CUDA_VISIBLE_DEVICES'] = '0'

os.environ['TF_FORCE_GPU_ALLOW_GROWTH'] = 'true'

import tensorflow as tf
import random
import numpy as np

print(f"TensorFlow git version is {tf.version.GIT_VERSION}")
print(f"TensorFlow version is {tf.version.VERSION}")

result={}
random.seed(4399)
np.random.seed(4399)
tf.random.set_seed(4399)
try:
    with tf.device('/CPU:0'):
        unamed_0 = tf.random.uniform([4, 3], dtype=tf.float32)
        unamed_1 = tf.random.uniform([3], minval=0, maxval=256, dtype=tf.int32)
        result["CPU_res"] = tf.nn.embedding_lookup(unamed_0, unamed_1, ) # this should results into error, and it does so
except Exception as e:
    result["CPU_err"] = str(e)
print(f"CPU input 1 is {unamed_0}")
print(f"CPU inptu 2 is {unamed_1}")
print(f"CPU result is {result}")

result = {}
random.seed(4399)
np.random.seed(4399)
tf.random.set_seed(4399)
try:
    with tf.device('/GPU:0'):
        unamed_0 = tf.random.uniform([4, 3], dtype=tf.float32)
        unamed_1 = tf.random.uniform([3], minval=0, maxval=256, dtype=tf.int32)
        result["GPU_res"] = tf.nn.embedding_lookup(unamed_0, unamed_1, ) # this should results into error, but it does not
except Exception as e:
    result["GPU_err"] = str(e)
print(f"GPU input 1 is {unamed_0}")
print(f"GPU inptu 2 is {unamed_1}")
print(f"GPU result is {result}")
```

I have also reproduced this bug in colab.

### Standalone code to reproduce the issue

```shell
https://colab.research.google.com/drive/1YqG7jHpMCLjrE8VqQlBSZtz_lRNgT4rU?usp=sharing
```


### Relevant log output

```shell
TensorFlow git version is v2.14.0-rc1-21-g4dacf3f368e
TensorFlow version is 2.14.0
CPU input 1 is [[0.35562682 0.13057125 0.9950352 ]
 [0.27489936 0.9225844  0.87167037]
 [0.44606853 0.45545673 0.3351872 ]
 [0.6091548  0.7782723  0.10762429]]
CPU inptu 2 is [144 161  30]
CPU result is {'CPU_err': '{{function_node __wrapped__GatherV2_device_/job:localhost/replica:0/task:0/device:CPU:0}} indices[0] = 144 is not in [0, 4) [Op:GatherV2] name: '}
GPU input 1 is [[0.35562682 0.13057125 0.9950352 ]
 [0.27489936 0.9225844  0.87167037]
 [0.44606853 0.45545673 0.3351872 ]
 [0.6091548  0.7782723  0.10762429]]
GPU inptu 2 is [144 161  30]
GPU result is {'GPU_res': <tf.Tensor: shape=(3, 3), dtype=float32, numpy=
array([[0., 0., 0.],
       [0., 0., 0.],
       [0., 0., 0.]], dtype=float32)>}
```
