Tensorflow terribly slow on Mac Studio M1 Ultra - problem with tensorflow-metal
Hi @francescosgarlata ,

I have replicated the issue. This problem observed with new optimizers earlier and now legacy optimizers also seems same problem. Since tensorflow-macos was built and maintained by Apple, could you please report the issue at [apple](https://developer.apple.com/forums/tags/tensorflow-metal/) dev forum ?

Also since this is keras related you can post this issue at [tf-keras](https://github.com/keras-team/tf-keras) repo also to track it by Keras team. 

Thanks! Hi just in case it is helpful, it should be a M1-Ultra-specific issue. I cannot reproduce it on M1 Pro. @francescosgarlata , tensorflow-metal is not released by Google, we only support official CPU support for M1 when installed through `pip install tensorflow`.

Could you please install tensorflow through pip and then install tensorflow-metal and see if that changes the performance.
If you notice the difference again, report the issue at https://developer.apple.com/forums/tags/tensorflow-metal/ This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62397">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62397">No</a>
 > Thank

Hi @sachinprasadhs and @francescosgarlata , we have seen this performance issue with moving to new Optimizer operations in Keras which are implemented 
https://github.com/tensorflow/tensorflow/issues/59438#issuecomment-1411414605

### Issue type

Performance

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

binary

### TensorFlow version

v2.13.0-rc2-7-g1cb1a030a62 2.13.0

### Custom code

No

### OS platform and distribution

Mac OS 13.5.2

### Mobile device

_No response_

### Python version

3.11

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

I have a Mac Studio M1 Ultra and I am trying to train a simple RNN to forecast time series.
When I run the code, tensorflow takes ~140s per epoch or 670ms/step.

If I uninstall tensorflow-metal, it takes only 20ms/step. But in this way I won't be able to use the GPU.

Why is this happening?

### Standalone code to reproduce the issue

```shell
import numpy as np
import keras 
import tensorflow as tf

def generate_time_series(batch_size, n_steps):
    freq1, freq2, offsets1, offsets2 = np.random.rand(4, batch_size, 1)
    time = np.linspace(0, 1, n_steps)
    series = 0.5 * np.sin((time - offsets1) * (freq1 * 10 + 10)) # wave 1
    series += 0.2 * np.sin((time - offsets2) * (freq2 * 20 + 20)) # + wave 2
    series += 0.1 * (np.random.rand(batch_size, n_steps) - 0.5) # + noise
    return series[..., np.newaxis].astype(np.float32)

np.random.seed(42)
n_steps = 50
series = generate_time_series(10000, n_steps + 1)

X_train, y_train = series[:7000, :n_steps], series[:7000, -1]
X_valid, y_valid = series[7000:9000, :n_steps], series[7000:9000, -1]
X_test, y_test = series[9000:, :n_steps], series[9000:, -1]

# Implementing simple RRN
np.random.seed(42)
tf.random.set_seed(42)

model = keras.models.Sequential([keras.layers.SimpleRNN(1, input_shape=[None, 1])])

optimizer = tf.keras.optimizers.legacy.Adam(learning_rate=0.005)
model.compile(loss="mse", optimizer=optimizer)
history = model.fit(X_train, y_train, epochs=20,
                    validation_data=(X_valid, y_valid))
```


### Relevant log output

```shell
Epoch 1/20

2023-11-14 20:58:05.842329: I metal_plugin/src/device/metal_device.cc:1154] Metal device set to: Apple M1 Ultra
2023-11-14 20:58:05.842356: I metal_plugin/src/device/metal_device.cc:296] systemMemory: 64.00 GB
2023-11-14 20:58:05.842361: I metal_plugin/src/device/metal_device.cc:313] maxCacheSize: 24.00 GB
2023-11-14 20:58:05.842396: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:303] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2023-11-14 20:58:05.842410: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:269] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
2023-11-14 20:58:06.172278: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:114] Plugin optimizer for device_type GPU is enabled.

219/219 [==============================] - 148s 676ms/step - loss: 0.4310 - val_loss: 0.2155
Epoch 2/20
219/219 [==============================] - 143s 651ms/step - loss: 0.1627 - val_loss: 0.1514
Epoch 3/20
219/219 [==============================] - 142s 649ms/step - loss: 0.1462 - val_loss: 0.1488
Epoch 4/20
219/219 [==============================] - 141s 644ms/step - loss: 0.1474 - val_loss: 0.1475
Epoch 5/20
219/219 [==============================] - 142s 649ms/step - loss: 0.1477 - val_loss: 0.1508
Epoch 6/20
219/219 [==============================] - 142s 649ms/step - loss: 0.1006 - val_loss: 0.0617
...
```
