tf.data train model worse than numpy data
@adamw-u Thank you for raising this issue!
Could you share the complete standalone code to reproduce this issue reported here. Could you please try to shuffle across the entire data in the training set and as a result the validation data would leak into the training set. Please let us know if that helps? Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61238">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61238">No</a>

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

tf 2.10

### Custom code

Yes

### OS platform and distribution

Linux Ubuntu 22.04

### Mobile device

_No response_

### Python version

3.8

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

12.0

### GPU model and memory

_No response_

### Current behavior?
**1. Used tf.data train model best val loss is 0.14, and the code is:**
![image](https://github.com/tensorflow/tensorflow/assets/15938790/36ab9474-3a88-47ba-9226-675f3dfeb11a)

```
train_dataset1 = tf.data.Dataset.from_tensor_slices((np.array(xtrain).reshape(-1,10*12,1),np.array(encoder_train),np.array(decoder_train)))
train_dataset2 = tf.data.Dataset.from_tensor_slices((np.array(ytrain)))
train_dataset = tf.data.Dataset.zip((train_dataset1, train_dataset2))
test_dataset1 = tf.data.Dataset.from_tensor_slices((np.array(xtest).reshape(-1,10*12,1),np.array(encoder_test),np.array(decoder_test)))
test_dataset2 = tf.data.Dataset.from_tensor_slices((np.array(ytest)))
test_dataset = tf.data.Dataset.zip((test_dataset1, test_dataset2))

BATCH_SIZE = 256
SHUFFLE_BUFFER_SIZE = 1000

train_dataset = train_dataset.shuffle(SHUFFLE_BUFFER_SIZE).batch(BATCH_SIZE)
test_dataset = test_dataset.batch(BATCH_SIZE)

model = Model(inputs = inputs, outputs = s2s)
model.compile(loss = 'mse', optimizer='adam', metrics=['mape'])
early_stopping = EarlyStopping(monitor='val_loss', patience=4, restore_best_weights=True)
model.fit(train_dataset, epochs=20, #batch_size=256, 
          validation_data = test_dataset, callbacks=[early_stopping])
```


**2. Used numpy array train model best val loss is 0.10, the code is:**
![image](https://github.com/tensorflow/tensorflow/assets/15938790/a9eec3fc-cdfe-4380-836d-7b047ab2580c)


```
model = Model(inputs = inputs, outputs = s2s)
model.compile(loss = 'mse', optimizer='adam', metrics=['mape'])
early_stopping = EarlyStopping(monitor='val_loss', patience=4, restore_best_weights=True)
model.fit([np.array(xtrain).reshape(-1,10*12,1),np.array(encoder_train),np.array(decoder_train)], np.array(ytrain), epochs=20, batch_size=256, 
          validation_data = ([np.array(xtest).reshape(-1,10*12,1),np.array(encoder_test),np.array(decoder_test)], np.array(ytest)), callbacks=[early_stopping])
```

**3. tf.data.Dataset.from_generator val loss worse more:**

### Standalone code to reproduce the issue

```shell
tf.data bugs at version 2.10
```


### Relevant log output

_No response_