Model performs differently on identical training and validation sets
Hi @gaikwadrahul8 -- have you been able to reproduce this issue? Just want to make sure it doesn't get lost Hi, @taylora-mitre 

Apologize for the delay and I was able to replicate the issue on Google colab with `Tensorflow==2.11` so I have added gist file for your reference [here](https://colab.research.google.com/gist/gaikwadrahul8/2360833ea46b4519353b601dc2da64dd/-59696.ipynb) and if training loss is converges to zero and still the validation loss is high this may indicate that the model is underfitting. It seems like mine and yours training loss and validation loss are different because you're generating random dataset in your code and you're using the same dataset as validation_data.

Just to confirm, you're expecting both training loss and validation loss should be same because you're using same identical training and validation dataset Am I Right ? or Am I misunderstanding something here so please correct me ? Thank you!

 @gaikwadrahul8 

Yes, that's correct. I am expecting the train and validation loss to be the same, since I'm using identical datasets. Or, if not perfectly identical, at least very close -- I am not expecting them to differ by several orders of magnitude as in your gist and mine @gaikwadrahul8 what are the next steps for this issue? Hi, @taylora-mitre 

Apologize for the delay and if you're following ResNet-18 model architecture with your customized functions and as you mentioned earlier training loss converging towards zero but still the validation loss high for random dataset with identical training and validation datasets it means that model is in underfitting but ideally we shouldn't use same training dataset for model validation.

If you want to build a reliable and robust model, you need to split your dataset into the training, validation, and test sets and If you don't, your results will be biased, and you'll end up with a false impression of better model accuracy.

The purpose of Validation set is our model training process is not a one-time process. We have to train multiple models by trying different combinations of hyperparameters. Then, we evaluate the performance of each model on the validation set. Therefore, the validation test is useful for hyperparameter tuning or selecting the best model out of different models

You're training your model only for 10 epochs so I don't think so you've built reliable, robust and generalise model so it will give you best result on validation dataset even though it's identical with training dataset in 10 epochs so could you please try for more number of epochs on GPU because the original ResNet-18 model architecture is complex and it has trained for some days with GPU, you can refer this [link](https://stackoverflow.com/questions/48051670/training-resnet-deep-neural-network-from-scratch), I hope it will help you to resolve your issue.

If I'm missing something here please let me know ? Thank you! @gaikwadrahul8 

It seems to me that you're missing the point of this issue entirely. I am not trying to build a robust, generalizable model; this is a toy example meant to demonstrate a specific technical issue.

Do you truly think that the model is underfitting here? If anything, the model should be drastically *over-fitting* since we only use one `(X, y)` pair. In fact, this is by design! I am demonstrating that the expected behavior (extreme overfitting, leading to zero training and validation loss) does not happen here. This is the core of the issue.

Again, let me say that I am NOT trying to build a robust, production-ready, generalizable model in this simple toy example. I am trying a prove a point, which I believe you may have missed. For that reason, splitting the dataset into unique train/validation/test sets is not applicable here; similarly, training for longer times is not applicable here. Hi, @taylora-mitre 

Apologize for the delayed response and I'm really sorry if I misunderstood your issue here so as you are using identical training and validation sets so you're expecting training and validation loss should be very close to zero or at least there shouldn't be difference by several orders of magnitude so (extreme overfitting, leading to zero training and validation loss) does not happening here, We'll have to dig more into this issue and we'll update you soon, Thank you for noticing this issue. Thank you! Yes, this is my issue. Thanks for the response. Let me know if you come up with anything for this! Hi, @taylora-mitre 

Apologize for the delayed response and I found [stack-overflow-answer-1](https://stackoverflow.com/questions/49165351/why-val-loss-is-different-from-training-loss-when-use-the-same-training-data-as) and [stack-overflow-answer-2](https://stackoverflow.com/questions/55444737/validation-loss-much-higher-than-training-loss) which may help you to understand why training and validation loss shouldn't be very close to zero or at least there shouldn't be difference by several orders of magnitude on identical training and validation datasets and in your case it's leading to `Data Leakage` where the Machine Learning Model is already aware of some part of test data after training.This causes the problem of overfitting. I hope it will help you to solve your issue. Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. For anyone who comes across this in the future, the issue is the `BatchNormalization` layers in the model creation. Replacing these with another kinds of normalization (such as e.g. `GroupNormalization`), or just removing them entirely, leads to the expected behavior in this toy example.

For real-life applications of ML, it is recommended to use an alternative kind of normalization rather than dropping it entirely. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59696">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59696">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

No

### Source

source

### Tensorflow Version

2.11.0

### Custom Code

Yes

### OS Platform and Distribution

_No response_

### Mobile device

_No response_

### Python version

_No response_

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current Behaviour?

Training a `tf.keras.models.Model` on a single, constant `(X, y)` does not behave as expected. The training loss converges to 0 as the model appears to learn the single training data element, but the validation loss remains high, despite the training and validation data being identical.

I am using a ResNet-18 model architecture (see https://arxiv.org/pdf/1711.09577.pdf or the attached gist) which uses several stacked convolutional layers to form a deep and robust model.


### Standalone code to reproduce the issue

```python
import numpy as np
import tensorflow as tf

# Build model
model = build_my_model() # see attached gist below for specific model build

# Generate training/validation data
input_data = np.random.random((batch_size,*imshape))
dummy_target = np.random.random((batch_size,3))

# Convert to tf.data.Dataset
ds1 = tf.data.Dataset.from_tensors(input_data)
ds2 = tf.data.Dataset.from_tensors(dummy_target)
dataset = tf.data.Dataset.zip((ds1,ds2)).repeat()
for img,target in dataset.take(1):
    print(img.shape, target)

# Fit model -- note training loss goes to 0, while validation loss does not
model.fit(
    dataset,
    epochs=10,
    steps_per_epoch=30,
    validation_data = dataset,
    validation_steps=3,
)

# Predict on the input data to show that the loss is still much larger than it should be
ypred = model.predict(input_data, verbose=0)
print(dummy_target, ypred, model.loss(ypred, dummy_target))
```

Reproducible gist in Google Colab
```shell
https://colab.research.google.com/gist/taylora-mitre/9ccdc0275f8d2119e33e311092c5334e/train-and-validation.ipynb
```


### Relevant log output

```
Epoch 1/10
30/30 [==============================] - 7s 224ms/step - loss: 5.9760e-05 - val_loss: 5.9585
Epoch 2/10
30/30 [==============================] - 7s 225ms/step - loss: 3.0297e-06 - val_loss: 2.6298
Epoch 3/10
30/30 [==============================] - 7s 228ms/step - loss: 1.2876e-07 - val_loss: 1.4914
Epoch 4/10
30/30 [==============================] - 7s 223ms/step - loss: 5.6785e-09 - val_loss: 1.0507
Epoch 5/10
30/30 [==============================] - 7s 229ms/step - loss: 2.6075e-10 - val_loss: 0.8550
Epoch 6/10
30/30 [==============================] - 7s 220ms/step - loss: 7.5849e-12 - val_loss: 0.7406
Epoch 7/10
30/30 [==============================] - 7s 218ms/step - loss: 5.1980e-13 - val_loss: 0.6433
Epoch 8/10
30/30 [==============================] - 7s 222ms/step - loss: 1.7708e-13 - val_loss: 0.5429
Epoch 9/10
30/30 [==============================] - 7s 221ms/step - loss: 3.3924e-13 - val_loss: 0.4408
Epoch 10/10
30/30 [==============================] - 7s 221ms/step - loss: 3.0570e-13 - val_loss: 0.3440
<keras.callbacks.History at 0x7f6960343eb0>
```

</details>