Inference significantly slower for Lenet5 models on both CPU and GPU when compared to other frameworks
@jf9123 - Can you try it on TF 2.12.  Also, would be able to share your script you are using to measure the performance differences across the frameworks? @sampathweb Here is a repo for reproduction sake; https://github.com/jf9123/lenet5_question

I use GPUtil to record the GPU VRAM usage.
### System information

-   **Have I written custom code (as opposed to using a stock example script
    provided in TensorFlow)**: Custom
-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux Ubuntu 18.04
-   **TensorFlow installed from (source or binary)**: binary
-   **TensorFlow version**: 2.5
-   **Python version**: 3.7.5
-   **GPU model and memory**:  NVIDIA Quadro RTX 4000 GPU with 8GB of VRAM
-  **CPU model**: Intel(R) Core(TM) i9-9900K CPU operating at 3.60GHz 
- **cuDNN version**: 7.6.5

### Describe the problem
Hello everybody.

I’ve been experimenting with different models and different frameworks, and I’ve noticed that  inference with a Lenet5 model on the MNIST dataset is significantly slower on Keras (28% slower on CPU and 108% slower on GPU) compared to the PyTorch and TensorFlow v1.X implementations. Note that I artificially increased the size of the testing set of the MNIST dataset by duplicating the set by 100 times for a total of 1 000 000 samples. I've done so so any difference in inference time may be clearer. Also note that this is observed even when the amount of memory Keras can reserve is limited (i.e., `tf.config.experimental.set_memory_growth(gpu, True)`)

Here are boxplots that showcase the performance of numerous Lenet5 models:

![infer time](https://user-images.githubusercontent.com/132307143/235554144-86b4e9e2-56c1-4029-b84a-0c838176444c.png)


Any ideas on what may be causing this?
