Gather out-of-range read will return different value with `jit_compile`
@YangChenyuan , I have replicated the reported behaviour on GPU runtime. Attached [gist](https://colab.sandbox.google.com/gist/SuryanarayanaY/ad9b6248a61404ed4572c5bb9261281d/61865_gpu.ipynb) for reference.

It seems there is inconsistency in tf.gather API , w.r.t jit compilation when applied out of range indices. We will have a look into it and update.

Thanks for reporting.
 To add more on CPU for out of range indices, exception will be raised. On GPU, for out of range indices '0' will be returned which will be intended.

The intended behaviour observed with `jit_compile=False`. With `jit_compile=True` we need to check why it's not consistent. Thanks for looking into this problem! It's not feasible to throw an error from XLA in this case, as gather could be fused with other operations. The implementation semantics assumes that all reads are in-range, and from what I recall it performs clamping others (@burmako should it be spec'd?) Yes, XLA's gather clamps out of bounds indices as discussed in https://github.com/openxla/stablehlo/blob/main/docs/spec.md#gather. Not sure about the semantics of TF's gather though. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61865">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61865">No</a>
 Why close this issue? Isn't it a problem that the semantic of tf.gather different from xla's? Tf's semantic is also very useful in some cases.

Since gather and scatter are symmetric operations, and scatter ignores out-of-bound indices, it is reasonable for gather to also ignore out-of-bound indices.

The implementation of Gather in tensorrt stores zero for OOB access, so the behavior of XLA is inconsistent with tensorrt.
https://docs.nvidia.com/deeplearning/tensorrt/api/c_api/classnvinfer1_1_1_i_gather_layer.html

@cheshire  @burmako  @x10000year if your proposal is to change HLO semantics to match that of TF, it's probably better to open a bug on openxla/stablehlo repo (or openxla/xla). @cheshire I think it's tf/jit's responsibility to correctly translate a tensorflow graph to a xla graph. If computation overhead is an issue for correct translation for the default behavior (store zero for OOB access), at least an option should be provided to specify the desired behavior for OOB access. In this case I agree it seems like a bridge issue/sharp corner, which should be at least documented.

@burmako any thoughts? A few thoughts:

1. The [TF Gather spec](https://www.tensorflow.org/api_docs/python/tf/gather) states the following:
- “The values must be in range [0, params.shape[axis])”
- “Indices are always validated on CPU, never validated on GPU.”
- “Caution: On CPU, if an out of bound index is found, an error is raised. On GPU, if an out of bound index is found, a 0 is stored in the corresponding output value.”

In some ways this is behaving as spec’ed, it looks like GPU specifics are hand-wavy. I think the doc could be changed to “OOB on GPU is undefined behavior” since it seems implementation specific as to what the GPU behavior is. Namely, I’m not positive on the meaning of JIT but I believe no-JIT dispatches directly to kernels and JIT is XLA compiled?

2. _“Should we sync HLO spec to TF?”_
 
I think it’s best to avoid this given the caveats of the TF spec. I.e. the spec as-is states “must be in range” but also “can be out of range”.

3. Can the TF/XLA bridge generate HLO that makes CPU/GPU behavior consistent?

It isn’t feasible to reliably detect OOB access statically in the bridge, since the bounds are an input argument. As such, runtime checking is needed, and we currently have no means of doing this.

There have been informal discussions on some sort of assertion ops in StableHLO which we could add this use case to the discussion (not sure if we have a ticket for this yet, will look). That said, perhaps raising errors from XLA is infeasible, and this wouldn’t work for this use case? Maybe something can be done with custom calls to validate bounds during runtime, but this would likely add a performance hit. How the gradient of scatter_add op works when OOB indices exist? OOB indices are valid in scatter_add specs. 

Is it better to provide an option to specify the behavior of OOB? like 1) store zeros; 2) undefine behavior. Can XLA compilation fuse gather with the operations of bound validation and storing zeros for OOB to avoid performance hit?

Personally I would expect all operators to avoid undefine behavior, as it is often source of troubles.
### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

source

### TensorFlow version

2.15.0-dev20230914

### Custom code

Yes

### OS platform and distribution

_No response_

### Mobile device

_No response_

### Python version

_No response_

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

When the index is out of range, `gather` will return different values with and without `jit_compile=True`.

The return values are expected to be consistent. Another solution is to throw errors in both cases.

### Standalone code to reproduce the issue

```shell
class Model(tf.keras.Model):

  def __init__(self):
    super(Model, self).__init__()

  @tf.function(jit_compile=True)
  def call(self, input):
    x = tf.gather(input, indices=[5, 8, 7, 16, 256, 123], axis=0) # 256 is out of range
    return x

m = Model()

input_shape = [256]
x1 = tf.ones(input_shape)

# Call model
y = m(x1)
print(y)
# tf.Tensor([1. 1. 1. 1. 1. 1.], shape=(6,), dtype=float32), with jit_compile=True
# tf.Tensor([1. 1. 1. 1. 0. 1.], shape=(6,), dtype=float32), without jit_compile=True
```


### Relevant log output

_No response_