The metrics values in fit and in evaluate do not match (tf+keras)
Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61945">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61945">No</a>
 Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61945">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61945">No</a>

### Issue type

Support

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

2.9

### Custom code

Yes

### OS platform and distribution

Linux Ubuntu

### Mobile device

_No response_

### Python version

3.8

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

I have the issue which I described under in detail. Might be it's a my mistake but I try to fix it not a one day.

I simplified it as much as possible fitting the model to show you exactly an issue.

Let it be not a pre-trained model ResNet50. And let's I provide a dataset by easy way where X_train and Y_train are the ndarray with correspond shapes [N, 32, 32, 3] and [N, class_num=3].

I set batch_size=128, shuffle=False. Now I fitting my model and after the last epoch I get the metric equals 0.976, BUT If I just make model.evaluate(X_train), where X_train is the same data I use in fit I get absolutely different value - 0.456. The question is - WHY?

That's logs of my fitting: 263/263 [==============================] - 7s 27ms/step - loss: 0.2063 - auc: 0.9899 - mc_f1: 0.9326

That's after evaluate: 1052/1052 [==============================] - 11s 9ms/step - loss: 0.6186 - auc: 0.9053 - mc_f1: 0.4993

To prevent questions - mc_f1 is a custom metric - averaged f1 calculated for each class separately for the multiclass case.

So the code:

    # Change last layers
    last_layer = model.output
    output = tf.keras.layers.Dense(classes, activation="softmax")(last_layer)
    model = tf.keras.models.Model(inputs=model.inputs, outputs=output)

    return model


mcf1 = MulticlassF1(num_classes=3)

resnet50 = get_resnet50_model([32, 32, 3], 3)
resnet50.compile(
    optimizer=Adam(),
    loss=tf.keras.losses.CategoricalCrossentropy(),
    metrics=[tf.metrics.AUC(name='auc'), mcf1],
)
```


Fitting:

```
deb_metric_callback = DebugMetricCallback(resnet50, (X_train, Y_train), mcf1)
resnet50.fit(
    X_train, Y_train,
    epochs=100,
    batch_size=128,
    # verbose=0,
    shuffle=False,
    callbacks=[deb_metric_callback],
)
```

I have the issue which I described under in detail. Might be it's a my mistake but I try to fix it not a one day.

I simplified it as much as possible fitting the model to show you exactly an issue.

Let it be not a pre-trained model ResNet50. And let's I provide a dataset by easy way where X_train and Y_train are the ndarray with correspond shapes [N, 32, 32, 3] and [N, class_num=3].

I set batch_size=128, shuffle=False. Now I fitting my model and after the last epoch I get the metric equals 0.976, BUT If I just make model.evaluate(X_train), where X_train is the same data I use in fit I get absolutely different value - 0.456. The question is - WHY?

That's logs of my fitting: 263/263 [==============================] - 7s 27ms/step - loss: 0.2063 - auc: 0.9899 - mc_f1: 0.9326

That's after evaluate: 1052/1052 [==============================] - 11s 9ms/step - loss: 0.6186 - auc: 0.9053 - mc_f1: 0.4993

To prevent questions - mc_f1 is a custom metric - averaged f1 calculated for each class separately for the multiclass case.

So the code:

The metric:
```

class MulticlassF1(tf.keras.metrics.Metric):

    def __init__(self, name='mc_f1', num_classes=None, **kwargs):
        super(MulticlassF1, self).__init__(name=name, **kwargs)
        self.__zero_support = tf.cast(1e-7, dtype=tf.float16)
        self.__cm = self.add_weight(name='fn', initializer='zeros', shape=[num_classes, num_classes])
        if num_classes is not None:
            self.__num_classes = num_classes

    def update_state(self, y_true, y_pred, sample_weight=None):
        y_pred = K.argmax(y_pred, axis=1)
        y_true = K.argmax(y_true, axis=1)
        m = tf.math.confusion_matrix(y_true, y_pred, num_classes=self.__num_classes, dtype=tf.float32)
        self.__cm.assign_add(m)

    def reset_state(self):
        self.__cm.assign(tf.zeros((self.__num_classes, self.__num_classes)))

    def result(self):
        denominator = 0
        m = self.__cm
        for i in range(m.shape[0]):
            tp = m[i, i]
            fn = K.sum(m[:, i]) - tp
            fp = K.sum(m[i, :]) - tp
            tn = K.sum(K.flatten(m)) - (tp + fn + fp)
            tp = K.cast(tp, dtype=tf.float16)
            tn = K.cast(tn, dtype=tf.float16)
            fp = K.cast(fp, dtype=tf.float16)
            fn = K.cast(fn, dtype=tf.float16)
            precision = tp / ((tp + fp) + self.__zero_support) + self.__zero_support
            recall = tp / (tf.cast(tp + fn, dtype=tf.float16) + self.__zero_support) + self.__zero_support
            denominator += (1 / precision + 1 / recall)
        f1_combined = K.cast(2 * m.shape[0] / denominator, dtype=tf.float32)
        return f1_combined
```
The model declaration

```
def get_resnet50_model(window_size, classes):
    # Model initialization
    model = tf.keras.applications.ResNet50V2(
        include_top=False,
        weights=None,
        input_tensor=None,
        input_shape=window_size,
        pooling='max',
        classes=classes,

    )

    # Change last layers
    last_layer = model.output
    output = tf.keras.layers.Dense(classes, activation="softmax")(last_layer)
    model = tf.keras.models.Model(inputs=model.inputs, outputs=output)

    return model


mcf1 = MulticlassF1(num_classes=3)

resnet50 = get_resnet50_model([32, 32, 3], 3)
resnet50.compile(
    optimizer=Adam(),
    loss=tf.keras.losses.CategoricalCrossentropy(),
    metrics=[tf.metrics.AUC(name='auc'), mcf1],
)
```
Fitting:

```
deb_metric_callback = DebugMetricCallback(resnet50, (X_train, Y_train), mcf1)
resnet50.fit(
    X_train, Y_train,
    epochs=100,
    batch_size=128,
    # verbose=0,
    shuffle=False,
    callbacks=[deb_metric_callback],
)
```
I tried to do some debug and make a CallBack:

```
class DebugMetricCallback(Callback):
    def __init__(self, test, metr):
        super().__init__()
        self.test = test
        self.metric = metr


    def on_epoch_end(self, epoch, logs=None):
        print(self.metric.result())
        x, y = self.test
        y_pred = self.model.predict(x)
        inline_measure = MulticlassF1(num_classes=3)
        inline_measure.update_state(y, y_pred)
        print(self.metric.result().numpy(), inline_measure.result().numpy())
```

Here I get absolutely different confusion matrices.

Also I thought that this is the problem with BatchNormalization, but I save and compare model before and after evaluate it wasn't changed.

### Standalone code to reproduce the issue

```shell
https://github.com/xxraytz/temp.git
```


### Relevant log output

_No response_