Ensemble provides either correct or (the same) incorrect inference results 
@joeyearsley,
Please, share colab link or simple standalone code with supporting files to reproduce the issue in our environment.It helps us in localizing the issue faster. Thank you! Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/65325">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/65325">No</a>

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

binary

### TensorFlow version

2.14

### Custom code

Yes

### OS platform and distribution

_No response_

### Mobile device

_No response_

### Python version

_No response_

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

I have an ensemble; I run the models sequentially. It's flaky in the results it returns; sometimes, one model gives the correct result of 0.37. Other times, it gives the incorrect result of 0.30 (never any other value).

I've tried turning on deterministic ops, but I still have this flaky issue; everything else stays the same.

I've tested on an old TitanX, T4 & A100 - this test is always flaky. 

I've tried debugging via the VLOG, but nothing jumps out; the only difference I can spot is for some reason, different cudnn Conv Ops might be chosen (not 100% from the logs).

Are there other suggestions on how to debug this issue? - I understand there might be decimal point changes but this swing is huge.  

### Standalone code to reproduce the issue

```shell
I can't share the models.
```


### Relevant log output

```shell
-
```
