Significantly higher RAM consumption for Keras Lenet-5 models (especially during inference) on Keras when compared to other frameworks
@jf9123,
Could you please provide the complete code to reproduce the issue and it helps to analyse the issue in an effective way. Thank you! @tilakrayal Here's the code; https://github.com/jf9123/lenet5_question @jf9123,
Could you please confirm if this is a duplicate of #60462. If yes, please feel free to move this issue to closed status. So that we can track the issue there. Thank you! @tilakrayal This issue is related to a higher RAM consumption during inference on CPU, while https://github.com/tensorflow/tensorflow/issues/60462 is about a slow inference time on both CPU and GPU. I'm not sure if the two are related. Thank you for opening this issue. Development of keras moved to another [repository](https://github.com/keras-team/keras/issues). 

Could you please post this issue on keras-team/keras [repo](https://github.com/keras-team/keras/issues).
To know more please refer:
https://discuss.tensorflow.org/t/keras-project-moved-to-new-repository-in-https-github-com-keras-team-keras/1999
Thank you!
 This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60494">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60494">No</a>

### System information

-   **Have I written custom code (as opposed to using a stock example script
    provided in TensorFlow)**: Custom
-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux Ubuntu 18.04
-   **TensorFlow installed from (source or binary)**: binary
-   **TensorFlow version**: 2.5
-   **Python version**: 3.7.5
-   **GPU model and memory**:  NVIDIA Quadro RTX 4000 GPU with 8GB of VRAM
-  **CPU model**: Intel(R) Core(TM) i9-9900K CPU operating at 3.60GHz 
- **cuDNN version**: 7.6.5


### Describe the problem
Hello everybody.

I’ve been experimenting with different models and different frameworks, and I’ve noticed that, when using CPU, RAM consumption when using Lenet5 models on the MNIST dataset is significantly higher on Keras compared to the PyTorch and TensorFlow v1.X implementations, with RAM consumption during inference standing out. Note that I artificially increased the size of the testing set of the MNIST dataset by duplicating the set by 100 times for a total of 1 000 000 samples. I've done so so any difference in inference time may be clearer.

This also might be related to the slowdown observed during inference on Keras (https://github.com/tensorflow/tensorflow/issues/60462)

Here are boxplots that showcase the RAM consumption of numerous Lenet5 models:

![Screenshot 2023-05-04 015324 - Copy](https://user-images.githubusercontent.com/132307143/236123835-7c2c46c6-f1f6-4fe7-9b12-bea3750cb3c5.png)


Any ideas on what may be causing this?
