I am noticing lower validation accuracy on my dataset between Tensorflow 2.4 and Tensorflow 2.9
@hlreicha,
I was facing a different issue while executing the given code. Kindly find the gist of it [here](https://colab.research.google.com/gist/tilakrayal/7eb3058c89de5c1776502b21f3efbabf/untitled1082.ipynb).

Also there might be below chances for the low validation accuracy.

**1.** You are trying to apply some sort of preprocessing (zero meaning, normalizing, etc.) to either your training set or validation set, but not the other.

**2.** If you built some layers that perform differently during training and inference from scratch, your model might be incorrectly implemented (e.g. are moving mean and moving standard deviation for batch normalization getting updated during training? 
If using dropout, are weights scaled properly during inference?). This might be the case if your code implements these things from scratch and does not use Tensorflow builtin functions.

**3.** Overfitting. I suspect that the other two options more likely in your specific situation as your validation accuracy might stuck at 50% for some epochs.

Using a random sample from your validation set: It means your validation set at each evaluation step is different, so is your validation-loss.
Using a weighted loss-function(which is used in case of highly imbalanced class-problems). At train step, you weigh your loss function based on class-weights, while at dev step you just calculate the un-weighted loss. In such case, though your network is stepping into convergence, you might see lots of fluctuations in validation loss after each train-step.  @tilakrayal 
Hi, thank you for your response. I can post more functioning code block later if it would help. The problem is that I am not getting low validation (remember, I am in the 90% range), I am getting lower validation score with TensorFlow 2.9 compared to TensorFlow 2.4. The code (loading image, preprocessing, model building, and etc) is all the same, it is the matter of me switching conda enviroments. 

1. I am not doing any preprocessing other than dividing the image by 255.0 and it is applied to both train/val sets. This never caused issues in TensorFlow 2.4
2. I am not using any custom layers, I import my EfficienetNet model, flatten it, and add the  final dense layer.
3. I have not experienced this behavior

I get these results after I ran my training script 5 times for each version.
- TensorFlow 2.4: ~97-98% Accuracy on the validation set.

- TensorFlow 2.9: ~93-95% Accuracy on the validation set

I should specify I set my random seed values as well. Did something change between both versions that causes EfficientNet to lose its performance? @hlreicha - Would you be able to create a complete script / Colab using standard datasets, for example in `tensorflow_datasets` https://github.com/tensorflow/datasets to replicate this issue? This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60399">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60399">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Support

### Have you reproduced the bug with TF nightly?

Yes

### Source

source

### Tensorflow Version

2.9 and 2.4

### Custom Code

Yes

### OS Platform and Distribution

Linux Ubuntu 20.04

### Mobile device

_No response_

### Python version

3.8

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current Behaviour?

I am trying to train an image classifier model using EfficientNetB1 on a custom dataset and I am trying out TensorFlow 2.4 and TensorFlow 2.9. I am using the exact same script with the same optimizer, augmentation, parameters, and dataset. I ran training 5 times and the results are around the same.

Results:

- TensorFlow 2.4: ~97-98% Accuracy on the validation set.

- TensorFlow 2.9: ~93-95% Accuracy on the validation set

More information: I am using Adam optimizer with 0.0001 lr, batch size of 16, using imagenet model weights, and categorical_crossentropy for my loss. I am using the same dataset on each version and I am using the same training script. I simply switch conda enviroments to TF 2.4 and 2.9.

Did something change between both versions that cause this discrepancy? Did the EfficientNet model weights change? Is the way the validation accuracy are calculated is different? Are the opimizers implementations are different?

I would appreciate your help and I would like some information on how to make it consistent between both versions. Thanks

### Standalone code to reproduce the issue

```shell
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0'
os.environ['TF_FORCE_GPU_ALLOW_GROWTH'] = 'true'
import tensorflow as tf
from tensorflow.keras.applications import EfficientNetB1

model_base = EfficientNetB1(weights='imagenet',include_top=False, input_shape=(image_size, image_size, 3))
model.add(model_base)
model._name = "EfficientNetB1"    
model.add(layers.Flatten())
model.add(tf.keras.layers.Dense(len(classes), activation='softmax'))

opt = Adam(learning_rate=1e-4)
model.compile(optimizer=opt, loss= 'categorical_crossentropy', metrics=['accuracy'])
```


### Relevant log output

_No response_</details>