gpu/cpu time performance
@RinnetenseiQ,
May i know how many CPU cores you have? The following are the reasons why the CPU performance is higher than the GPU  in some of the model training cases.
- For small models there will be no performance improvements when using GPU and you are going to benefit from GPU only when you have `large weight matrices`.

-  For small matrices it will be faster on CPU because it has higher **frequency** and this overhead of invoking kernels and copying is only worsens the situation. Try multiplying matrices of these shapes **(10000, 10000)x(10000x10000)**. You will see that the **GPU is much faster**.

- Try increasing number of parameters in your model. There is an overhead when invoking GPU kernels and copying data to/from GPU. 

Also the overhead of invoking GPU kernels, and copying data to and from GPU, is very high. For operations on models with very little parameters it is not worth using GPU since frequency of CPU cores is much higher. If you compare matrix multiplication, you will see that for large matrices GPU outperforms CPU significantly. [Reference](https://github.com/tensorflow/tensorflow/issues/31654). Thank you!

 This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59963">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59963">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Performance

### Have you reproduced the bug with TF nightly?

No

### Source

source

### Tensorflow Version

2.10

### Custom Code

No

### OS Platform and Distribution

Windows 10

### Mobile device

_No response_

### Python version

3.9

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

cuDNN = 8.1.0.77

### GPU model and memory

nVidia GTX 1050, 4GB GRAM

### Current Behaviour?

```shell
I have noticed that training on cpu faster than gpu. Note that I have tried same on RTX 3090 TI and got similar results 

GPU performance:

2023-03-12 13:45:41.870315: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-03-12 13:45:42.381082: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 2787 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1050, pci bus id: 0000:01:00.0, compute capability: 6.1
WARNING:tensorflow:Please add `keras.layers.InputLayer` instead of `keras.Input` to Sequential model. `keras.Input` is intended to be used by Functional model.
Epoch 1/100
3125/3125 [==============================] - 10s 3ms/step - loss: 0.0495
Epoch 2/100
3125/3125 [==============================] - 10s 3ms/step - loss: 0.0014
Epoch 3/100
3125/3125 [==============================] - 10s 3ms/step - loss: 0.0013
Epoch 4/100
3125/3125 [==============================] - 11s 4ms/step - loss: 0.0013
Epoch 5/100
3125/3125 [==============================] - 11s 4ms/step - loss: 0.0012
Epoch 6/100
3125/3125 [==============================] - 11s 4ms/step - loss: 0.0011
```

CPU performance:
```
2023-03-12 13:48:54.873454: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:Please add `keras.layers.InputLayer` instead of `keras.Input` to Sequential model. `keras.Input` is intended to be used by Functional model.
Epoch 1/100
3125/3125 [==============================] - 4s 1ms/step - loss: 0.0511
Epoch 2/100
3125/3125 [==============================] - 4s 1ms/step - loss: 0.0019
Epoch 3/100
3125/3125 [==============================] - 4s 1ms/step - loss: 0.0013
Epoch 4/100
3125/3125 [==============================] - 4s 1ms/step - loss: 0.0013
Epoch 5/100
3125/3125 [==============================] - 4s 1ms/step - loss: 0.0013
Epoch 6/100
3125/3125 [==============================] - 4s 1ms/step - loss: 0.0012
```

### Standalone code to reproduce the issue

```
import numpy as np
from tensorflow.keras.utils import to_categorical
from tensorflow.python.keras.layers import Dense, Input
from tensorflow.python.keras.models import Model, Sequential

def build_std_func_model(n_inputs):
    inp = Input(n_inputs)
    h = Dense(100, activation="relu")(inp)
    h = Dense(100, activation="relu")(h)
    h = Dense(100, activation="relu")(h)
    h = Dense(100, activation="relu")(h)
    output = Dense(n_inputs, activation="linear")(h)
    return Model(inputs=inp, outputs=output)

def build_std_seq_model(n_inputs):
    model = Sequential()

    model.add(Input(n_inputs))
    model.add(Dense(100, activation="relu"))
    model.add(Dense(100, activation="relu"))
    model.add(Dense(100, activation="relu"))
    model.add(Dense(100, activation="relu"))
    model.add(Dense(n_inputs, activation="linear"))
    return model

def make_func_dataset(n_examples=30000, n_linears=15, n_categorical=4):
    linear = np.random.randn(n_examples, n_linears)
    categorical = [np.random.choice([0, 1, 2], size=n_examples) for _ in range(n_categorical)]
    categorical = [to_categorical(item) for item in categorical]
    return [linear, *categorical]

def run_simplest():
    X = make_func_dataset(n_examples=10**5, n_linears=30, n_categorical=0)[0]
    # model = build_std_func_model(30)
    model = build_std_seq_model(30)
    model.compile(optimizer="adam", loss="mean_squared_error")
    model.fit(X, X, epochs=100)

if __name__ == '__main__':
    # tf.config.set_visible_devices([], 'GPU')
    # run_functional()
    run_simplest()
```


### Relevant log output

_No response_</details>