Performance drop with tensorflow 2.13
@nhuet I tried to replicate the issue on colab using TF [v2.12](https://colab.research.google.com/gist/sushreebarsa/e498716b10932a7d557ab3a3a9fccb76/61363-2-12.ipynb) and [2.13](https://colab.research.google.com/gist/sushreebarsa/3ec5e639c34bb5c3d965c531fe1090b3/untitled813.ipynb#scrollTo=UpjWS8k2G3ms), could you please have a look at the attached gists and confirm the same?
Thank you! It seems to show the issue (even though only 3* slower, this is still abnormal) @nhuet thank you for the quick response
@sachinprasadhs Could you please have a look at this. Thank you! Hi @nhuet, thank you for bringing up this issue! Would you be able to try to replicate this with the `keras_core` (https://keras.io/keras_core/) package as this will supersede current Keras. Thank you! Hi @grasskin. You are right I should have definitely tried this. And this works very well indeed (even better than keras2.12 + tf 2.12). Here are the new timings:
- keras 2.12.0 + tensorflow 2.12.1: 7.8 s
- keras 2.13.1 + tensorflow 2.13.0: 24.1 s
- keras-core 0.1.3 + tensorflow 2.12.1: 0.4 s
- keras-core 0.1.3 + tensorflow 2.13.0: 0.3 s

(and even increasing the number of iteration, there is not noticeable differences between keras-core + tf2.12 or keras-core + tf2.13)

Thus, the "bug" came from keras 2.13, and the performance is way better with keras-core than with previous keras 2.12.
Thanks a lot! Hi @nhuet , Thanks for confirming your findings with the keras-core. 
I'm glad that keras-core + tensorflow gave the good performance result.

For latest package details, please visit https://pypi.org/project/keras-core/

Feel free to close the issue. Thanks! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61363">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61363">No</a>

### Issue type

Performance

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

binary

### TensorFlow version

v2.13.0-rc2-7-g1cb1a030a62 2.13.0

### Custom code

Yes

### OS platform and distribution

Debian GNU/Linux 12 (bookworm)

### Mobile device

_No response_

### Python version

3.9

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

I notice a big performance drop between tensorflow 2.12 (CPU) and tensorflow 2.13 (CPU). With the last release (and also with tf-nightly '2.14.0-dev20230724') it takes *4 times* longer to perform a simple sum. It is between keras inputs though some I am not sure if this is directly related to tensorflow or if it comes from keras.

See code below for a very simple example. Note that this is with tensorflow CPU only.
The timings are:
- for tensorflow 2.12.1 + keras 2.12.0: **5.3 s**
- for tensorflow 2.13.0 + keras 2.13.1: **24 s**
- for tensorflow 2.14.0-dev20230724 + keras  2.14.0.dev2023072407: 26 s

### Standalone code to reproduce the issue

```shell
import time

from tensorflow.keras import Input


number_of_executions = 3000

x = Input((1,))
y = Input((1,))

start = time.time()
for i in range(number_of_executions):
    x + y
duration = time.time() - start
print(f"Duration: {duration:.1f}")
```


### Relevant log output

_No response_