Tensorflow optimizer.apply_gradients is very slow.
@anshkumar 
This issue seems to be a keras related issue. Please post this issue on [keras-team/keras repo.](https://github.com/keras-team/keras/issues)
To know more see;
https://discuss.tensorflow.org/t/keras-project-moved-to-new-repository-in-https-github-com-keras-team-keras/1999.

Thank you! Posted issue to [keras](https://github.com/keras-team/tf-keras/issues/238) @anshkumar 
This issue will be triaged on [Keras repo](https://github.com/keras-team/tf-keras/issues/238). So, Could you please close this issue ?

Thank you ! Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59961">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59961">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

Yes

### Source

binary

### Tensorflow Version

2.13

### Custom Code

No

### OS Platform and Distribution

Ubuntu 20.04

### Mobile device

_No response_

### Python version

3.10

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current Behaviour?

```shell
I'm using a custom training loop for training a model made using EfficientNetV2 with biFPN. I found that `optimizer.apply_gradients` was running very slow. It took around 1.7 seconds to `apply_gradients` while model was taking only 0.1 seconds. I tried to replicate this using smaller example, so followed tutorial [here](https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch). I found a similar issue here also.  `apply_gradients` here also is taking twice the time compared to model+loss. Is it fine to have such behavior? I tried a different version of tensorflow, but found the same behavior.
```


### Standalone code to reproduce the issue

```shell
The code used can be found here:
https://colab.research.google.com/github/tensorflow/docs/blob/snapshot-keras/site/en/guide/keras/writing_a_training_loop_from_scratch.ipynb
```


### Relevant log output

```shell
Model+Loss  0.027350187301635742
Grad  0.027795791625976562
Apply Grad  0.056526899337768555
```
</details>