Accuracy metric appears to behave incorrectly with multiple outputs
I discovered that this does not occur if you replace the Lambda layers with two single neuron dense outputs. I guess you might still want to investigate why the Lambda layers make it behave like that. I suppose it could cause issues if someone happened to need one at the output. Otherwise, feel free to close it. Hi @jtolladay ,

The Metrics will be resetted normally after each epoch till then the states will be getting updated which is default behaviour.Here as we have 800 samples and the accuracy will be multiple of 1/800 i.e.0.00125 and for validation_data also it will be multiples of 1/200 and it's been followed in logs. Correct me if I am wrong here.

You can also refer to the low level handling of Metrics section of custom training loop from scratch from the [documentation](https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch#low-level_handling_of_metrics). However with custom training loop we can change the behaviour as per our requirement.

Thanks! Please see my original post:

"In the log you can see that the accuracies on the training set (with 80 batches of size 10) go up in steps of **0.0**125 (1/80). The validation set (containing 20 batches of size 10) go up in steps of **0.0**5 (1/20). It must be somehow calculating the fraction of the batches that are "correct" instead of the number of correct predictions out of all the batches."

Like I said, I fixed the problem by replacing the size 2 output and lambda layers with just two separate dense layers with 1 neuron each. I assume the issue with the lambda layers was that they make the shape of the tensor unusual when they split the outputs, so the metric was averaging along the wrong dimension or something. Whether that's an issue with the lambda layers or my implementation, I am not sure. If it is not a bug then again, feel free to close it. Hi @jtolladay ,

I am curious to explore this case more. I have stored the training checkpoints to a history object. I tried to multiply the accuracies(both accuracy and val_accuracy) with total no of samples(800) and total no of batches(80) to check which will output a whole number. 

For the case with Lambda layers for accuracy both total samples and batches produces whole numbers and multiples of 10.
But for validation accuracies both fail to return whole number for some cases.

Whereas the case without Lamda layers ( replaced with Dense layer) I found accuracy is exact multiple of no of samples(800) but not batches(80) which indicates that accuracy is calculated on no of samples only but not batch.
Where as for validation accuracy neither no of samples(200) nor batches(20) produces whole numbers.

It seems for Lambda layer both no of samples and no of batches producing whole numbers but at the same time it is important to note that for all cases the result is multiple of batches only.This needs to be checked. Also for validation accuracies none of them producing the whole numbers which is also an issue to look into.

I am sharing the [gist](https://colab.research.google.com/gist/SuryanarayanaY/451158dd070f1db50176598fbe1c9d80/60510.ipynb) for demonstration of same. Please go through and share your opinion and correct me if anything is wrong here.




 Hi @SuryanarayanaY,

I tried this code:

```
import numpy as np
import tensorflow as tf

tf.keras.utils.set_random_seed(45)

# define the data
x = np.random.random(1000)
y1 = np.random.randint(0, 2, 1000)
y2 = np.random.randint(0, 2, 1000)

# define the models
def model_lambda():
    input = tf.keras.layers.Input(shape=(1,))
    model = tf.keras.layers.Dense(64, activation='relu')(input)
    model = tf.keras.layers.Dense(2, activation='sigmoid')(model)
    out1 = tf.keras.layers.Lambda(lambda x: x[..., 0], name='out1')(model)
    out2 = tf.keras.layers.Lambda(lambda x: x[..., 1], name='out2')(model)
    model = tf.keras.models.Model(input, [out1, out2])
    return model

def model_dense():
    input = tf.keras.layers.Input(shape=(1,))
    model = tf.keras.layers.Dense(64, activation='relu')(input)
    out1 = tf.keras.layers.Dense(1, activation='sigmoid', name='out1')(model)
    out2 = tf.keras.layers.Dense(1, activation='sigmoid', name='out2')(model)
    model = tf.keras.models.Model(input, [out1, out2])
    return model

for model_func, model_name in zip([model_lambda, model_dense], ['lambda model', 'dense model']):
    # configure losses and metrics for each output
    losses = {'out1': 'binary_crossentropy', 'out2': 'binary_crossentropy'}
    weights = {'out1': 1.0, 'out2': 1.0}
    metrics = {'out1': 'accuracy', 'out2': 'accuracy'}
    model = model_func()
    model.compile(loss=losses, loss_weights=weights, optimizer='adam', metrics=metrics)

    # fit
    history = model.fit(x, [y1, y2], batch_size=10, epochs=20, verbose=0, validation_split=0.2)

    # analyse
    print('----------', model_name, '----------')
    out1, out2 = model.predict(x)
    print('out1 shape:', out1.shape, ', out2 shape:', out2.shape)
    for metric in ['out1_accuracy', 'out2_accuracy', 'val_out1_accuracy', 'val_out2_accuracy']:
        accuracies = np.array(history.history[metric])
        diffs = np.diff(np.sort(accuracies))
        min_diff = np.min(diffs[diffs != 0])
        print(metric, 'is out of', np.round(1 / min_diff))
```

and got this as the log output:

---------- lambda model ----------
out1 shape: (1000,) , out2 shape: (1000,)
out1_accuracy is out of 80.0
out2_accuracy is out of 80.0
val_out1_accuracy is out of 20.0
val_out2_accuracy is out of 20.0
---------- dense model ----------
out1 shape: (1000, 1) , out2 shape: (1000, 1)
out1_accuracy is out of 800.0
out2_accuracy is out of 800.0
val_out1_accuracy is out of 200.0
val_out2_accuracy is out of 200.0

Which would appear to confirm what I was finding. I printed the shape of the outputs because I believe it is this different shape which is causing the accuracy to be predicted differently. As I said in my previous post, the issue here is probably that the reshaping done in the lambda layers is done incorrectly, rather than the lambda layers themselves being the issue. I don't have more time to test this now, but I think it's likely my code, rather than tensorflow that is the issue. I had a few minutes and confirmed what I thought to be the issue. Modifying the lambda model to be like so:

```
def model_lambda():
    input = tf.keras.layers.Input(shape=(1,))
    model = tf.keras.layers.Dense(64, activation='relu')(input)
    model = tf.keras.layers.Dense(2, activation='sigmoid')(model)
    out1 = tf.keras.layers.Lambda(lambda x: x[..., 0])(model)
    out1 = tf.keras.layers.Reshape((1,), name='out1')(out1)
    out2 = tf.keras.layers.Lambda(lambda x: x[..., 1])(model)
    out2 = tf.keras.layers.Reshape((1,), name='out2')(out2)
    model = tf.keras.models.Model(input, [out1, out2])
    return model
```

Results in the correct output that we would expect:

---------- lambda model ----------
out1 shape: (1000, 1) , out2 shape: (1000, 1)
out1_accuracy is out of 800.0
out2_accuracy is out of 800.0
val_out1_accuracy is out of 200.0
val_out2_accuracy is out of 200.0
---------- dense model ----------
out1 shape: (1000, 1) , out2 shape: (1000, 1)
out1_accuracy is out of 800.0
out2_accuracy is out of 800.0
val_out1_accuracy is out of 200.0
val_out2_accuracy is out of 200.0 @jtolladay ,

Yeah it seems the difference is due to your custom model with different output shapes. Thanks for confirming the same. Shall we marked it as resolved now. please spare some time for closing if resolved for you. Thanks! Happy to close as it is more an issue with my code rather than with tensorflow itself. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60510">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60510">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

No

### Source

source

### Tensorflow Version

2.7.0

### Custom Code

Yes

### OS Platform and Distribution

Windows 10

### Mobile device

_No response_

### Python version

3.9.9

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current Behaviour?

In the example below, the training set is 800 records, validation set is 200 records and batch size is 10. My understanding of the accuracy metric is that it should go up in steps of 1/[number of records] between 0 and 1. It's result should be the fraction of all individual predictions that were correct. As the description of the accuracy metric on tensorflow's website: "Calculates how often predictions equal labels."

However, in the log you can see that the accuracies on the training set (with 80 batches of size 10) go up in steps of 0.0125 (1/80). The validation set (containing 20 batches of size 10) go up in steps of 0.05 (1/20). It must be somehow calculating the fraction of the batches that are "correct" instead of the number of correct predictions out of all the batches.

### Standalone code to reproduce the issue

```shell
import numpy as np
import tensorflow as tf

tf.keras.utils.set_random_seed(1)

# define the data
x = np.random.random(1000)
y1 = np.random.randint(0, 2, 1000)
y2 = np.random.randint(0, 2, 1000)

# define the model
input = tf.keras.layers.Input(shape=(1,))
model = tf.keras.layers.Dense(64, activation='relu')(input)
model = tf.keras.layers.Dense(2, activation='sigmoid')(model)
out1 = tf.keras.layers.Lambda(lambda x: x[..., 0], name='out1')(model)
out2 = tf.keras.layers.Lambda(lambda x: x[..., 1], name='out2')(model)
model = tf.keras.models.Model(input, [out1, out2])

# configure losses and metrics for each output
losses = {'out1': 'binary_crossentropy', 'out2': 'binary_crossentropy'}
weights = {'out1': 1.0, 'out2': 1.0}
metrics = {'out1': 'accuracy', 'out2': 'accuracy'}
model.compile(loss=losses, loss_weights=weights, optimizer='adam', metrics=metrics)

# fit
model.fit(x, [y1, y2], batch_size=10, epochs=10, verbose=2, validation_split=0.2)
```


### Relevant log output

```shell
2023-05-05 14:27:37.862813: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-05-05 14:27:38.294938: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1242 MB memory:  -> device: 0, 
name: NVIDIA T500, pci bus id: 0000:01:00.0, compute capability: 7.5
Epoch 1/10
80/80 - 1s - loss: 1.3891 - out1_loss: 0.6953 - out2_loss: 0.6938 - out1_accuracy: 0.1125 - out2_accuracy: 0.1625 - val_loss: 1.3861 - val_out1_loss: 0.6933 - val_out2_loss: 0.6929 
- val_out1_accuracy: 0.1500 - val_out2_accuracy: 0.0500 - 1s/epoch - 18ms/step
Epoch 2/10
80/80 - 0s - loss: 1.3866 - out1_loss: 0.6931 - out2_loss: 0.6935 - out1_accuracy: 0.0625 - out2_accuracy: 0.0375 - val_loss: 1.3843 - val_out1_loss: 0.6918 - val_out2_loss: 0.6925 
- val_out1_accuracy: 0.2000 - val_out2_accuracy: 0.1000 - 313ms/epoch - 4ms/step
Epoch 3/10
80/80 - 0s - loss: 1.3872 - out1_loss: 0.6938 - out2_loss: 0.6934 - out1_accuracy: 0.1125 - out2_accuracy: 0.1250 - val_loss: 1.3837 - val_out1_loss: 0.6916 - val_out2_loss: 0.6920 
- val_out1_accuracy: 0.2000 - val_out2_accuracy: 0.0500 - 331ms/epoch - 4ms/step
Epoch 4/10
80/80 - 0s - loss: 1.3865 - out1_loss: 0.6929 - out2_loss: 0.6935 - out1_accuracy: 0.1125 - out2_accuracy: 0.1125 - val_loss: 1.3834 - val_out1_loss: 0.6916 - val_out2_loss: 0.6918 
- val_out1_accuracy: 0.2000 - val_out2_accuracy: 0.1000 - 335ms/epoch - 4ms/step
Epoch 5/10
80/80 - 0s - loss: 1.3859 - out1_loss: 0.6929 - out2_loss: 0.6930 - out1_accuracy: 0.0750 - out2_accuracy: 0.1125 - val_loss: 1.3832 - val_out1_loss: 0.6917 - val_out2_loss: 0.6915 
- val_out1_accuracy: 0.2000 - val_out2_accuracy: 0.1000 - 339ms/epoch - 4ms/step
Epoch 6/10
80/80 - 0s - loss: 1.3861 - out1_loss: 0.6931 - out2_loss: 0.6930 - out1_accuracy: 0.1125 - out2_accuracy: 0.0625 - val_loss: 1.3831 - val_out1_loss: 0.6919 - val_out2_loss: 0.6913 
- val_out1_accuracy: 0.2000 - val_out2_accuracy: 0.1500 - 333ms/epoch - 4ms/step
Epoch 7/10
80/80 - 0s - loss: 1.3860 - out1_loss: 0.6934 - out2_loss: 0.6926 - out1_accuracy: 0.1000 - out2_accuracy: 0.0625 - val_loss: 1.3830 - val_out1_loss: 0.6920 - val_out2_loss: 0.6911 
- val_out1_accuracy: 0.2500 - val_out2_accuracy: 0.1000 - 335ms/epoch - 4ms/step
Epoch 8/10
80/80 - 0s - loss: 1.3860 - out1_loss: 0.6929 - out2_loss: 0.6930 - out1_accuracy: 0.1000 - out2_accuracy: 0.1250 - val_loss: 1.3826 - val_out1_loss: 0.6918 - val_out2_loss: 0.6908 
- val_out1_accuracy: 0.2000 - val_out2_accuracy: 0.2000 - 342ms/epoch - 4ms/step
Epoch 9/10
80/80 - 0s - loss: 1.3861 - out1_loss: 0.6930 - out2_loss: 0.6931 - out1_accuracy: 0.0625 - out2_accuracy: 0.1250 - val_loss: 1.3824 - val_out1_loss: 0.6917 - val_out2_loss: 0.6907 
- val_out1_accuracy: 0.2000 - val_out2_accuracy: 0.1500 - 336ms/epoch - 4ms/step
Epoch 10/10
80/80 - 0s - loss: 1.3857 - out1_loss: 0.6930 - out2_loss: 0.6927 - out1_accuracy: 0.0750 - out2_accuracy: 0.1250 - val_loss: 1.3824 - val_out1_loss: 0.6919 - val_out2_loss: 0.6905 
- val_out1_accuracy: 0.2000 - val_out2_accuracy: 0.1000 - 349ms/epoch - 4ms/step
```
</details>