Discrepancy in training accuracy for CNN on MNIST dataset between Apple Silicon and colab
colab training output:
```
Epoch 1/20
938/938 [==============================] - 9s 8ms/step - loss: 0.3648 - accuracy: 0.8979 - val_loss: 0.2319 - val_accuracy: 0.9322
Epoch 2/20
938/938 [==============================] - 4s 4ms/step - loss: 0.1963 - accuracy: 0.9444 - val_loss: 0.1679 - val_accuracy: 0.9508
Epoch 3/20
938/938 [==============================] - 4s 4ms/step - loss: 0.1470 - accuracy: 0.9582 - val_loss: 0.1314 - val_accuracy: 0.9609
Epoch 4/20
938/938 [==============================] - 3s 3ms/step - loss: 0.1187 - accuracy: 0.9663 - val_loss: 0.1143 - val_accuracy: 0.9648
Epoch 5/20
938/938 [==============================] - 3s 3ms/step - loss: 0.0998 - accuracy: 0.9720 - val_loss: 0.1046 - val_accuracy: 0.9674
Epoch 6/20
938/938 [==============================] - 4s 4ms/step - loss: 0.0866 - accuracy: 0.9757 - val_loss: 0.0940 - val_accuracy: 0.9706
Epoch 7/20
938/938 [==============================] - 4s 4ms/step - loss: 0.0766 - accuracy: 0.9787 - val_loss: 0.0872 - val_accuracy: 0.9735
Epoch 8/20
938/938 [==============================] - 3s 4ms/step - loss: 0.0686 - accuracy: 0.9811 - val_loss: 0.0834 - val_accuracy: 0.9744
Epoch 9/20
938/938 [==============================] - 3s 3ms/step - loss: 0.0617 - accuracy: 0.9831 - val_loss: 0.0809 - val_accuracy: 0.9746
Epoch 10/20
938/938 [==============================] - 4s 5ms/step - loss: 0.0562 - accuracy: 0.9844 - val_loss: 0.0786 - val_accuracy: 0.9746
Epoch 11/20
938/938 [==============================] - 4s 4ms/step - loss: 0.0511 - accuracy: 0.9861 - val_loss: 0.0760 - val_accuracy: 0.9764
Epoch 12/20
938/938 [==============================] - 3s 3ms/step - loss: 0.0467 - accuracy: 0.9872 - val_loss: 0.0768 - val_accuracy: 0.9757
Epoch 13/20
938/938 [==============================] - 3s 4ms/step - loss: 0.0426 - accuracy: 0.9886 - val_loss: 0.0761 - val_accuracy: 0.9760
Epoch 14/20
938/938 [==============================] - 4s 5ms/step - loss: 0.0396 - accuracy: 0.9895 - val_loss: 0.0731 - val_accuracy: 0.9757
Epoch 15/20
938/938 [==============================] - 3s 4ms/step - loss: 0.0361 - accuracy: 0.9906 - val_loss: 0.0706 - val_accuracy: 0.9780
Epoch 16/20
938/938 [==============================] - 4s 4ms/step - loss: 0.0335 - accuracy: 0.9913 - val_loss: 0.0705 - val_accuracy: 0.9781
Epoch 17/20
938/938 [==============================] - 3s 3ms/step - loss: 0.0310 - accuracy: 0.9922 - val_loss: 0.0697 - val_accuracy: 0.9777
Epoch 18/20
938/938 [==============================] - 5s 5ms/step - loss: 0.0286 - accuracy: 0.9932 - val_loss: 0.0692 - val_accuracy: 0.9770
Epoch 19/20
938/938 [==============================] - 4s 4ms/step - loss: 0.0265 - accuracy: 0.9938 - val_loss: 0.0689 - val_accuracy: 0.9780
Epoch 20/20
938/938 [==============================] - 4s 4ms/step - loss: 0.0245 - accuracy: 0.9943 - val_loss: 0.0673 - val_accuracy: 0.9787
``` mac training output:
```
Epoch 1/20
 10/938 [..............................] - ETA: 5s - loss: 1.9703 - accuracy: 0.3406  2023-11-08 20:41:34.161484: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:114] Plugin optimizer for device_type GPU is enabled.
938/938 [==============================] - ETA: 0s - loss: 0.4114 - accuracy: 0.88262023-11-08 20:41:38.610013: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:114] Plugin optimizer for device_type GPU is enabled.
938/938 [==============================] - 5s 5ms/step - loss: 0.4114 - accuracy: 0.8826 - val_loss: 0.3143 - val_accuracy: 0.9107
Epoch 2/20
938/938 [==============================] - 4s 5ms/step - loss: 0.3101 - accuracy: 0.9133 - val_loss: 0.3021 - val_accuracy: 0.9165
Epoch 3/20
938/938 [==============================] - 5s 5ms/step - loss: 0.2998 - accuracy: 0.9161 - val_loss: 0.2989 - val_accuracy: 0.9155
Epoch 4/20
938/938 [==============================] - 4s 5ms/step - loss: 0.2983 - accuracy: 0.9166 - val_loss: 0.3063 - val_accuracy: 0.9144
Epoch 5/20
938/938 [==============================] - 4s 5ms/step - loss: 0.3038 - accuracy: 0.9149 - val_loss: 0.3682 - val_accuracy: 0.8932
Epoch 6/20
938/938 [==============================] - 4s 5ms/step - loss: 0.3150 - accuracy: 0.9122 - val_loss: 0.3739 - val_accuracy: 0.8932
Epoch 7/20
938/938 [==============================] - 4s 5ms/step - loss: 0.3397 - accuracy: 0.9064 - val_loss: 0.4372 - val_accuracy: 0.8834
Epoch 8/20
938/938 [==============================] - 4s 5ms/step - loss: 0.3819 - accuracy: 0.8998 - val_loss: 0.4067 - val_accuracy: 0.8965
Epoch 9/20
938/938 [==============================] - 4s 5ms/step - loss: 0.4602 - accuracy: 0.8902 - val_loss: 0.8379 - val_accuracy: 0.8237
Epoch 10/20
938/938 [==============================] - 4s 5ms/step - loss: 0.6021 - accuracy: 0.8800 - val_loss: 1.0389 - val_accuracy: 0.8353
Epoch 11/20
938/938 [==============================] - 4s 5ms/step - loss: 0.7349 - accuracy: 0.8748 - val_loss: 1.0670 - val_accuracy: 0.8420
Epoch 12/20
938/938 [==============================] - 4s 5ms/step - loss: 0.9430 - accuracy: 0.8689 - val_loss: 1.2278 - val_accuracy: 0.8605
Epoch 13/20
938/938 [==============================] - 4s 5ms/step - loss: 1.1816 - accuracy: 0.8641 - val_loss: 1.4626 - val_accuracy: 0.8539
Epoch 14/20
938/938 [==============================] - 4s 5ms/step - loss: 1.5326 - accuracy: 0.8634 - val_loss: 1.5809 - val_accuracy: 0.8603
Epoch 15/20
938/938 [==============================] - 4s 5ms/step - loss: 1.9404 - accuracy: 0.8633 - val_loss: 1.8023 - val_accuracy: 0.8647
Epoch 16/20
938/938 [==============================] - 4s 5ms/step - loss: 2.4296 - accuracy: 0.8607 - val_loss: 3.9961 - val_accuracy: 0.8077
Epoch 17/20
938/938 [==============================] - 4s 5ms/step - loss: 3.2530 - accuracy: 0.8572 - val_loss: 3.5723 - val_accuracy: 0.8546
Epoch 18/20
938/938 [==============================] - 4s 5ms/step - loss: 4.0268 - accuracy: 0.8612 - val_loss: 5.4699 - val_accuracy: 0.8384
Epoch 19/20
938/938 [==============================] - 4s 5ms/step - loss: 5.1973 - accuracy: 0.8587 - val_loss: 7.9400 - val_accuracy: 0.7539
Epoch 20/20
938/938 [==============================] - 4s 5ms/step - loss: 6.6197 - accuracy: 0.8568 - val_loss: 14.9388 - val_accuracy: 0.7909
``` @N33MO,
Thank you for reporting the issue. We are currently investigating the issue, and I kindly request some time to thoroughly analyze the problem in order to offer a resolution and also could you please confirm is this happening in tf v2.14 as well? Thank you! Hi @tilakrayal,
The issue still show on v2.14: https://colab.research.google.com/drive/1bQe1F4PO2G-kaLWvDQcnqzLMsddoUux9
Here's the package installed on the Mac machine.
<img width="709" alt="image" src="https://github.com/tensorflow/tensorflow/assets/27294839/b2f1b65a-8517-40b4-a783-851b07947746">
 @N33MO @tilakrayal Seems like the issue is happening when using the Apple Silicon GPU as I get nearly identical results as colab on my M1 Max if I hide the GPU using the below code(tf v2.14):
`hw = tf.config.get_visible_devices()`
`tf.config.set_visible_devices(hw[0])`

Mac output:

> [PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'), PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
WARNING:absl:At this time, the v2.11+ optimizer `tf.keras.optimizers.SGD` runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at `tf.keras.optimizers.legacy.SGD`.
Epoch 1/20
938/938 [==============================] - 1s 824us/step - loss: 0.3648 - accuracy: 0.8979 - val_loss: 0.2319 - val_accuracy: 0.9322
Epoch 2/20
938/938 [==============================] - 1s 745us/step - loss: 0.1963 - accuracy: 0.9443 - val_loss: 0.1677 - val_accuracy: 0.9510
Epoch 3/20
938/938 [==============================] - 1s 743us/step - loss: 0.1471 - accuracy: 0.9583 - val_loss: 0.1313 - val_accuracy: 0.9605
Epoch 4/20
938/938 [==============================] - 1s 736us/step - loss: 0.1187 - accuracy: 0.9662 - val_loss: 0.1142 - val_accuracy: 0.9647
Epoch 5/20
938/938 [==============================] - 1s 752us/step - loss: 0.0998 - accuracy: 0.9719 - val_loss: 0.1045 - val_accuracy: 0.9673
Epoch 6/20
938/938 [==============================] - 1s 744us/step - loss: 0.0866 - accuracy: 0.9757 - val_loss: 0.0937 - val_accuracy: 0.9708
Epoch 7/20
938/938 [==============================] - 1s 743us/step - loss: 0.0766 - accuracy: 0.9787 - val_loss: 0.0872 - val_accuracy: 0.9733
Epoch 8/20
938/938 [==============================] - 1s 745us/step - loss: 0.0685 - accuracy: 0.9812 - val_loss: 0.0835 - val_accuracy: 0.9746
Epoch 9/20
938/938 [==============================] - 1s 731us/step - loss: 0.0617 - accuracy: 0.9831 - val_loss: 0.0809 - val_accuracy: 0.9750
Epoch 10/20
938/938 [==============================] - 1s 737us/step - loss: 0.0561 - accuracy: 0.9845 - val_loss: 0.0788 - val_accuracy: 0.9750
Epoch 11/20
938/938 [==============================] - 1s 732us/step - loss: 0.0510 - accuracy: 0.9860 - val_loss: 0.0755 - val_accuracy: 0.9763
Epoch 12/20
938/938 [==============================] - 1s 752us/step - loss: 0.0467 - accuracy: 0.9872 - val_loss: 0.0766 - val_accuracy: 0.9763
Epoch 13/20
938/938 [==============================] - 1s 741us/step - loss: 0.0426 - accuracy: 0.9886 - val_loss: 0.0762 - val_accuracy: 0.9761
Epoch 14/20
938/938 [==============================] - 1s 745us/step - loss: 0.0395 - accuracy: 0.9894 - val_loss: 0.0727 - val_accuracy: 0.9761
Epoch 15/20
938/938 [==============================] - 1s 745us/step - loss: 0.0360 - accuracy: 0.9906 - val_loss: 0.0707 - val_accuracy: 0.9779
Epoch 16/20
938/938 [==============================] - 1s 745us/step - loss: 0.0334 - accuracy: 0.9913 - val_loss: 0.0707 - val_accuracy: 0.9773
Epoch 17/20
938/938 [==============================] - 1s 740us/step - loss: 0.0310 - accuracy: 0.9921 - val_loss: 0.0700 - val_accuracy: 0.9773
Epoch 18/20
938/938 [==============================] - 1s 747us/step - loss: 0.0286 - accuracy: 0.9933 - val_loss: 0.0698 - val_accuracy: 0.9769
Epoch 19/20
938/938 [==============================] - 1s 742us/step - loss: 0.0265 - accuracy: 0.9937 - val_loss: 0.0690 - val_accuracy: 0.9783
Epoch 20/20
938/938 [==============================] - 1s 754us/step - loss: 0.0246 - accuracy: 0.9942 - val_loss: 0.0674 - val_accuracy: 0.9792 I'm encountering not exactly the same but I think related issue. 
I use MacBook Air with M2. Running the same code on CPU and on GPU (with `tensorflow-metal`) gives different results. On GPU it causes to model training fail. 
Please see all the details in related [issue](https://github.com/keras-team/tf-keras/issues/140#issuecomment-1824331706).
 @tilakrayal I tested the code on v2.15 as well, the issue still exists and is not resolved as mentioned [here](https://github.com/keras-team/tf-keras/issues/140).
 @romanilchyshyn Regarding the running time of CPU vs GPU, in order to take advantage of GPU you should have sufficient load. For basic learning CPU is faster. I have reproduced the same issue with tf2.15 as well using M2 MacBook Air GPU training... increasing loss, compared to CPU mode where everything seems fine... @sachinprasadhs,
I was able to reproduce the issue on tensorflow v2.14, v2.15 and also on the Mac-OS M1. Kindly find the gist of it [here](https://colab.research.google.com/gist/tilakrayal/713489165dd196cc354db8d64589ec02/tf_cnn_mnist_example_colab.ipynb) and the logs below.

Epoch 11/20
938/938 [==============================] - 5s 5ms/step - loss: 0.7349 - accuracy: 0.8748 - val_loss: 1.0670 - val_accuracy: 0.8420
Epoch 12/20
938/938 [==============================] - 5s 5ms/step - loss: 0.9430 - accuracy: 0.8689 - val_loss: 1.2278 - val_accuracy: 0.8605
Epoch 13/20
938/938 [==============================] - 5s 5ms/step - loss: 1.1816 - accuracy: 0.8641 - val_loss: 1.4626 - val_accuracy: 0.8539
Epoch 14/20
938/938 [==============================] - 5s 5ms/step - loss: 1.5326 - accuracy: 0.8634 - val_loss: 1.5809 - val_accuracy: 0.8603
Epoch 15/20
938/938 [==============================] - 5s 5ms/step - loss: 1.9404 - accuracy: 0.8633 - val_loss: 1.8023 - val_accuracy: 0.8647
Epoch 16/20
938/938 [==============================] - 5s 5ms/step - loss: 2.4296 - accuracy: 0.8607 - val_loss: 3.9961 - val_accuracy: 0.8077
Epoch 17/20
938/938 [==============================] - 5s 5ms/step - loss: 3.2530 - accuracy: 0.8572 - val_loss: 3.5723 - val_accuracy: 0.8546
Epoch 18/20
938/938 [==============================] - 5s 5ms/step - loss: 4.0268 - accuracy: 0.8612 - val_loss: 5.4699 - val_accuracy: 0.8384
Epoch 19/20
938/938 [==============================] - 5s 5ms/step - loss: 5.1973 - accuracy: 0.8587 - val_loss: 7.9400 - val_accuracy: 0.7539
Epoch 20/20
938/938 [==============================] - 5s 5ms/step - loss: 6.6197 - accuracy: 0.8568 - val_loss: 14.9388 - val_accuracy: 0.7909 I am also seeing radically different (and worse) results with `tensorflow-metal` using the GPU on an M3 Max than I do just with the CPU.

It is worth noting that the problems get worse as the network gets deeper. If I test using a simple model like:
```python
model = tf.keras.models.Sequential([
  tf.keras.Input(name="x", shape=(28, 28, 1)),
  tf.keras.layers.Flatten(),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10)
])
```
then I get similar results to the ones posted above. If I try a more complex model such as:
```python
model2 = tf.keras.models.Sequential([
    tf.keras.Input(name="x", shape=(28, 28, 1)),
    tf.keras.layers.Conv2D(32, (3, 3), activation='relu'),
    tf.keras.layers.MaxPooling2D((2, 2)),
    tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
    tf.keras.layers.MaxPooling2D((2, 2)),
    tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(10, name="y")
])
```
then with the CPU I'm getting 99.5% accuracy after 10 epochs whereas with the GPU using Metal the accuracy starts on the 85% range but plummets to around 11% after 10 epochs and doesn't recover. > I am also seeing radically different (and worse) results with `tensorflow-metal` 2.1.1 using the GPU on an M3 Max than I do just with the CPU.
> 
> It is worth noting that the problems get worse as the network gets deeper. If I test using a simple model like:
> 
> ```python
> model = tf.keras.models.Sequential([
>   tf.keras.Input(name="x", shape=(28, 28, 1)),
>   tf.keras.layers.Flatten(),
>   tf.keras.layers.Dense(128, activation='relu'),
>   tf.keras.layers.Dropout(0.2),
>   tf.keras.layers.Dense(10)
> ])
> ```
> 
> then I get similar results to the ones posted above. If I try a more complex model such as:
> 
> ```python
> model2 = tf.keras.models.Sequential([
>     tf.keras.Input(name="x", shape=(28, 28, 1)),
>     tf.keras.layers.Conv2D(32, (3, 3), activation='relu'),
>     tf.keras.layers.MaxPooling2D((2, 2)),
>     tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
>     tf.keras.layers.MaxPooling2D((2, 2)),
>     tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
>     tf.keras.layers.Flatten(),
>     tf.keras.layers.Dense(64, activation='relu'),
>     tf.keras.layers.Dropout(0.2),
>     tf.keras.layers.Dense(10, name="y")
> ])
> ```
> 
> then with the CPU I'm getting 99.5% accuracy after 10 epochs whereas with the GPU using Metal the accuracy starts on the 85% range but plummets to around 11% after 10 epochs and doesn't recover.

Regarding `model2` as mentioned above:

For me, on Apple M1 Pro w/ Ventura, using GPU, val_loss stays pegged at 5.0967 across 15 epochs, while w/o GPU it goes from 13.5264 - 14.4612.

 There appears to be something significantly wrong with the back propagation implementation on Metal. Running inference on snapshots from elsewhere delivers identical results with useful performance improvements. When training categorisation models that happily converge on the CPU, when you try using Metal the accuracy degenerates towards not much better than guesswork. The time it takes to degenerate depends on the model depth. 

As it stands, training on Metal is simply broken and `tensorflow-metal` is only really useful for inference. Same here.
M3 Pro 
TF 2.15.0
tensorflow-metal 1.1.0

Very annoing...
### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

2.13

### Custom code

Yes

### OS platform and distribution

macOS 14.1

### Mobile device

_No response_

### Python version

_No response_

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

Running a simple example training a CNN using MNIST dataset usnig tensorflow, the training accuracy/loss yield very different results.

### Standalone code to reproduce the issue

```shell
colab output: https://colab.research.google.com/drive/1jE9qmdaGXbC9lt69kUUzzzJq7dZJnVJ8?usp=sharing
mac output: https://colab.research.google.com/drive/1mr9JPqX7XzVVevdyueiGpQu8Hc7eBHw1
```


### Relevant log output

_No response_