Tensorflow freezes during training on Mac Studio
Try to update TensorFlow version with "pip install --upgrade tensorflow" I did, and the problem still appears with tensorflow `2.12`. Hi @hovavalon ,

It seems the issue is Mac specific and Mac M1 related issues has to be addressed in the Apple metal plugin forum.

But I observed earlier there is some issues related to the optimizer. Can you please replace the optimizer with legacy optimizer as below and test it with TF2.9, TF2.11, TF2.12 and let us know the outcome.

`opt = tf.keras.optimizers.legacy.SGD()` Hi @SuryanarayanaY, I am already using the legacy optimizer, at some point I got an error recommending that I do so. @hovavalon ,

I tried to replicate the issue on Mac with tf-nightly(2.14.0-dev20230515) with `epochs=100` with the same test code you provided and its running fine.Please refer to logs below. 

```
(tf-metal) suryanarayanay-macbookpro:~ suryanarayanay$ python Downloads/60800.py
2.14.0-dev20230515
Metal device set to: Apple M1 Pro

systemMemory: 16.00 GB
maxCacheSize: 5.33 GB

Epoch 1/100
10/10 [==============================] - 1s 3ms/step - loss: 7.0479
Epoch 2/100
10/10 [==============================] - 0s 3ms/step - loss: 4.2836
Epoch 3/100
10/10 [==============================] - 0s 2ms/step - loss: 2.6105
Epoch 4/100
10/10 [==============================] - 0s 3ms/step - loss: 1.5956
Epoch 5/100
10/10 [==============================] - 0s 2ms/step - loss: 0.9820
Epoch 6/100
10/10 [==============================] - 0s 3ms/step - loss: 0.6105
Epoch 7/100
10/10 [==============================] - 0s 3ms/step - loss: 0.3856
Epoch 8/100
10/10 [==============================] - 0s 3ms/step - loss: 0.2490
Epoch 9/100
10/10 [==============================] - 0s 2ms/step - loss: 0.1656
Epoch 10/100
10/10 [==============================] - 0s 3ms/step - loss: 0.1148
Epoch 11/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0843
Epoch 12/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0653
Epoch 13/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0537
Epoch 14/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0463
Epoch 15/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0416
Epoch 16/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0385
Epoch 17/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0364
Epoch 18/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0348
Epoch 19/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0337
Epoch 20/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0328
Epoch 21/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0320
Epoch 22/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0313
Epoch 23/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0306
Epoch 24/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0300
Epoch 25/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0295
Epoch 26/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0290
Epoch 27/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0285
Epoch 28/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0280
Epoch 29/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0275
Epoch 30/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0271
Epoch 31/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0266
Epoch 32/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0261
Epoch 33/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0257
Epoch 34/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0253
Epoch 35/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0249
Epoch 36/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0245
Epoch 37/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0241
Epoch 38/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0237
Epoch 39/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0234
Epoch 40/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0230
Epoch 41/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0226
Epoch 42/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0223
Epoch 43/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0220
Epoch 44/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0216
Epoch 45/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0213
Epoch 46/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0211
Epoch 47/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0208
Epoch 48/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0205
Epoch 49/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0202
Epoch 50/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0199
Epoch 51/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0197
Epoch 52/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0194
Epoch 53/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0191
Epoch 54/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0189
Epoch 55/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0187
Epoch 56/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0184
Epoch 57/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0182
Epoch 58/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0180
Epoch 59/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0178
Epoch 60/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0176
Epoch 61/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0173
Epoch 62/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0172
Epoch 63/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0170
Epoch 64/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0168
Epoch 65/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0166
Epoch 66/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0164
Epoch 67/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0163
Epoch 68/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0161
Epoch 69/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0159
Epoch 70/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0158
Epoch 71/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0156
Epoch 72/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0155
Epoch 73/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0153
Epoch 74/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0152
Epoch 75/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0150
Epoch 76/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0149
Epoch 77/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0148
Epoch 78/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0147
Epoch 79/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0145
Epoch 80/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0144
Epoch 81/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0143
Epoch 82/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0142
Epoch 83/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0140
Epoch 84/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0139
Epoch 85/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0138
Epoch 86/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0137
Epoch 87/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0136
Epoch 88/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0136
Epoch 89/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0134
Epoch 90/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0134
Epoch 91/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0132
Epoch 92/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0132
Epoch 93/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0131
Epoch 94/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0130
Epoch 95/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0129
Epoch 96/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0128
Epoch 97/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0128
Epoch 98/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0127
Epoch 99/100
10/10 [==============================] - 0s 2ms/step - loss: 0.0126
Epoch 100/100
10/10 [==============================] - 0s 3ms/step - loss: 0.0125
(tf-metal) suryanarayanay-macbookpro:~ suryanarayanay$ 
```
Can you try with latest nightly version and please confirm the behaviour.

Also please confirm you have followed the Apple [metal](https://developer.apple.com/metal/tensorflow-plugin/) plugin instructions correctly.Please confirm you have installed tensorflow-metal and try retesting the code in a Fresh environment.

Thanks!

 I have followed the instructions given in the link when installing the application.
Even on my computer 100 short epochs are not necessarily enough to replicate (even though at some times it is enough).  Hi @hovavalon ,

Thanks for confirmation. I tried with 1000 epohs and executed 3 times successfully without the reported behaviour.Please refer to attached logs below.

[#60800_logs.txt](https://github.com/tensorflow/tensorflow/files/11742629/60800_logs.txt)

Could you try with fresh environment and with tf-nightly version.

Thanks!
 This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60800">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60800">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

Yes

### Source

binary

### Tensorflow Version

tf2.9

### Custom Code

No

### OS Platform and Distribution

MacOS Ventura 13.4/13.1

### Mobile device

Mac Studio

### Python version

3.10

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

Apple M1

### Current Behaviour?

When training a model on my Mac Studio, from time to time it seems that the training freezes, and an epoch takes much more time than usual, from x10 to x200.
I have tried contacting Apple's support, and they are clueless about this phenomena.
I also went through recent Mac related issues in this repository and found nothing relevant.

Here is an example of the phenomena as seen in the training's output:
```
28/28 [==============================] - 5s 195ms/step - loss: 6.2743e-04 - accuracy: 0.0113 - val_loss: 3.4481e-04 - val_accuracy: 0.0147
Epoch 597/1000
28/28 [==============================] - 61s 2s/step - loss: 8.0337e-04 - accuracy: 0.0055 - val_loss: 3.4126e-04 - val_accuracy: 0.0244
Epoch 598/1000
28/28 [==============================] - 5s 195ms/step - loss: 7.9272e-04 - accuracy: 0.0101 - val_loss: 3.3659e-04 - val_accuracy: 0.0208
```





### Standalone code to reproduce the issue

```shell
import tensorflow as tf
import numpy as np

# Generate random training data
np.random.seed(0)
x_train = np.random.rand(100, 1)
y_train = 3 * x_train + 2 + np.random.randn(100, 1) * 0.1

# Define the neural network architecture
model = tf.keras.models.Sequential([
    tf.keras.layers.Dense(1, input_shape=(1,))
])

# Compile the model
model.compile(optimizer='sgd', loss='mean_squared_error')

# Train the model
model.fit(x_train, y_train, epochs=1000, batch_size=10)
```

(Thank ChatGPT for the minimal working example)
```


### Relevant log output

```shell
Metal device set to: Apple M1 Ultra

systemMemory: 128.00 GB
maxCacheSize: 48.00 GB

2023-06-07 13:46:01.901374: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2023-06-07 13:46:01.901665: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
2023-06-07 13:46:01.992101: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
Epoch 1/1000
2023-06-07 13:46:02.108687: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
10/10 [==============================] - 0s 3ms/step - loss: 14.7499
Epoch 2/1000
10/10 [==============================] - 0s 10ms/step - loss: 9.1414
...
Epoch 33/1000
10/10 [==============================] - 0s 13ms/step - loss: 0.0382
Epoch 34/1000
10/10 [==============================] - 6s 716ms/step - loss: 0.0374
Epoch 35/1000
10/10 [==============================] - 0s 5ms/step - loss: 0.0367
...
```
</details>