Output tensor inconsistencies across different TF versions
@DLumi Could you please check whether you are using the same TF version for training and inference?
Please provide a simple standalone code to replicate the issue reported here. 
Thank you! @sushreebarsa I'm sorry, but did you even read what I wrote in the issue template? 
There you can find as simple (and standalone) code as it can get. But you will have to switch environments to replicate that, given that the bug occurs on different TF versions.

The model in question was indeed saved (and later inferenced) on the same TF version. But I believe, it should also work if you save the model on another version, like 2.10. Did not check this one, as even nailing this down + filing the issue took quite a lot of time. So hope you can look into that. As per the result which is observed in the comparison, 

```
Max absolute difference: 5.9604645e-08
Max relative difference: 2.3390216e-07 
```

It is in the range of` 1e-07`, which should be fine, considering the floating point precision errors. Thank you! @sachinprasadhs for the attached tensors the error is indeed _relatively_ low. 
But depending on the inputs and the model, the error can get up to 1e-5, which is a bit too much to write it off as FP precision error. Hi @DLumi, could you try saving with .npz instead of pickle? Pickle saving is known to be different across python version and can lead to inconsistencies. Could you try using a CPU environment instead of a GPU environment? That might also reduce inconsistencies. We also suggest that you do not switch python versions for this test so we can isolate the potential issue to Tensorflow. Thank you!  This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62175">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62175">No</a>

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

binary

### TensorFlow version

2.12.1 / 2.14.0

### Custom code

No

### OS platform and distribution

Win 10 22H2

### Mobile device

_No response_

### Python version

3.8.0 / 3.11.5

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

0) Save some simple model (does not really matter which one)
1) Activate Py 3.8 + TF 2.12.1 environment
2) Run inference on some random tensor with predetermined seed
3) Save the output tensor to a pickle file
4) Activate Py 3.11.5 + TF 2.14 environment
5) Run inference on the same random tensor with predetermined seed
6) Load previous output tensor which we pickled earlier
7) Compare two outputs with `np.testing.assert_allclose` with default `rtol` of `1e-07`
8) Get an error that the tensors mismatch
9) ???
10) PROFIT

For your convenience, I will attach the model and the pickled tensors. The code I used to produce it all will be down below.
https://drive.google.com/drive/folders/1LWeotexKYGlzu2TtqbwN7ErgW_X_4C_4

### Standalone code to reproduce the issue

#### On Py 3.8 + TF 2.12.1 environment:

```python
out_dir = Path(r'')

image_input = tf.keras.Input(shape=[224, 224, 3])

conv1 = tf.keras.layers.SeparableConv2D(32, (3, 3))(image_input)
conv1 = tf.keras.layers.BatchNormalization()(conv1)
conv1 = tf.keras.layers.ReLU()(conv1)
pool1 = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(conv1)

conv2 = tf.keras.layers.SeparableConv2D(64, (3, 3))(pool1)
conv2 = tf.keras.layers.BatchNormalization()(conv2)
conv2 = tf.keras.layers.ReLU()(conv2)
pool2 = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(conv2)

out = tf.keras.layers.Dense(4, activation='softmax')(pool2)

model = tf.keras.Model(image_input, out)

model.compile(loss='categorical_crossentropy',
              optimizer=tf.keras.optimizers.Adam(),
              metrics=['categorical_accuracy'])

model.save(out_dir / 'model_tf_12_1')
```

Then load the saved_model, and save the prediction
```python
model_1 = tf.saved_model.load(out_dir / 'model_tf_12_1')

np.random.seed(123) # predetermined seed
random_tensor = np.random.random([1, 224, 224, 3]).astype('float32')

pred_1 = model_1(random_tensor)

with open(out_dir / 'pred_1', mode='wb') as f:
    pickle.dump(pred_1, f)
```

#### On Py 3.11.5 + TF 2.14.0 environment:

```python
model_1 = tf.saved_model.load(out_dir / 'model_tf_12_1')

np.random.seed(123) # predetermined seed
random_tensor = np.random.random([1, 224, 224, 3]).astype('float32')

pred_2 = model_1(random_tensor)

with open(out_dir / 'pred_1', mode='rb') as f:
    pred_1 = pickle.load(f)

np.testing.assert_allclose(pred_1, pred_2)
```

See the output of `np.testing.assert_allclose` below


### Relevant log output

```shell
AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0

Mismatched elements: 753 / 11664 (6.46%)
Max absolute difference: 5.9604645e-08
Max relative difference: 2.3390216e-07
 x: array([[[[0.259732, 0.241168, 0.242284, 0.256816],
         [0.25886 , 0.242881, 0.243037, 0.255222],
         [0.256769, 0.243074, 0.243309, 0.256848],...
 y: array([[[[0.259732, 0.241168, 0.242284, 0.256816],
         [0.25886 , 0.242881, 0.243037, 0.255222],
         [0.256769, 0.243074, 0.243309, 0.256848],...
```
