Performance using prefetch does not improve!
@pietroorlandi,
Thank you for raising the issue. I tried to reproduce the issue on tensorflow v2.12 and facing the issue with the dataset. Could you please provide the TFRECORD_FILE_TRAIN dataset/tfrecord to analyse the issue in an effective way. Also could you please provide the GPU device information where you are trying. Kindly find the gist of it [here](https://colab.research.google.com/gist/tilakrayal/6c6d16e7b29a057b7c0133d6eb18ea2b/untitled1080.ipynb). Thank you! @tilakrayal thanks for your reply!
How can I upload my tfrecords dataset in the gist? I tried with "upload in session storage space", it is a file called _train.tfrecords_.
The tests that I made to say that prefetch optimization doesn't change the performance were done with two GPU: RTX2080 and RTX2060 Super I too have noticed the same problem with tensorflow 2.10 and tensorflow 2.11: I don't see any perfomance gain with prefetch optimization compared to using a parallel version of map function (both setting manually the num_parallel_calls and using AUTOTUNE).
(as GPU I'm using a NVIDIA GeForce GTX 1050Ti) @tilakrayal Is there any news?
I saw that, as it mentioned [here](https://www.reddit.com/r/MachineLearning/comments/a08fx6/d_why_tfdata_is_so_much_better_than_feed_dict_and/) it seems that _prefetch_ does not overlap the computation between the CPU and GPU and you don't have the behavior that while GPU train _batch i_ the CPU processes _batch i+1_. It seems that prefetch function overlap data reading and data processing. Is this right?
Always [here](https://www.reddit.com/r/MachineLearning/comments/a08fx6/d_why_tfdata_is_so_much_better_than_feed_dict_and/) I saw that exist a function called _prefetch_to_device_ that would allow the overlap of computation between CPU and GPU. I tried it but again it does not provide any improvement.
Thank you in advance for the answers that will be provided to me
<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Performance

### Have you reproduced the bug with TF nightly?

No

### Source

source

### Tensorflow Version

2.11.1

### Custom Code

Yes

### OS Platform and Distribution

Ubuntu 22.04.2 LTS

### Mobile device

_No response_

### Python version

3.9.16

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

12.0

### GPU model and memory

_No response_

### Current Behaviour?

Hi all, the problem I have is about _tf.data_ and mainly about the fact that _prefetch_ optimization does not improve perfomance in any case (and it almost seems that it is not used).
Specifically, I did the tests using different environments (with different GPUs and different CPUs), using different versions of tensorflow (I used tf-2.10.0 and tf-2.11.0), using different batch-size, using different data-processing/data-augmentation operations, using different parameters of _num_parallel_calls_ (with fixed number and with AUTOTUNE) and the results are the same: I have no advantage using prefetching.
This seems strange to me, because at least theoretically a speed-up should be there. I used Cifar10 as the dataset, and the data are initially stored in a TFRecordDataset. PS: I already see this [guide](https://www.tensorflow.org/guide/data_performance)
Thank you in advance 

### Standalone code to reproduce the issue

```shell

import numpy as np
import tensorflow as tf
import tensorflow.keras as keras
import tensorflow.keras.layers as layers
from keras.utils.layer_utils import count_params
import time
from datetime import datetime

def get_model_data_augmentation_CPU():
    model = tf.keras.Sequential([
      layers.Conv2D(64, 3, activation='relu'),
      layers.MaxPooling2D(),
      layers.Dropout(0.1),
      layers.Conv2D(128, 3, activation='relu'),
      layers.MaxPooling2D(),
      layers.Dropout(0.1),
      layers.Conv2D(128, 3, activation='relu'),
      layers.MaxPooling2D(),
      layers.Dropout(0.2),
      layers.Flatten(),
      layers.Dense(256, activation='relu'),
      layers.Dropout(0.3),
      layers.Dense(10)
    ])
    adam_opt = keras.optimizers.Adam(learning_rate=0.001)
    model.compile(optimizer = adam_opt,
                  loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
                  metrics=['accuracy'])
    return model

def parse_image_function(example_proto):
    image_feature_description = {
        'image': tf.io.FixedLenFeature([], tf.string),
        'label': tf.io.FixedLenFeature([], tf.int64)
    }
    features = tf.io.parse_single_example(
                  example_proto, image_feature_description)
    image = tf.io.decode_raw(features['image'], tf.float32)
    image = tf.reshape(image, [32, 32, 3])
    label = tf.cast(features['label'], tf.int64)
    return image, label

# Set Memory Growth of tensorflow equals true
gpus = tf.config.list_physical_devices('GPU')
if gpus:
    try:
        # Currently, memory growth needs to be the same across GPUs
        for gpu in gpus:
            tf.config.experimental.set_memory_growth(gpu, True)
        logical_gpus = tf.config.list_logical_devices('GPU')
        print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
    except RuntimeError as e:
        # Memory growth must be set before GPUs have been initialized
        print(e)

NUM_CLASSES = 10 # Cifar10 has 10 classes
BATCH_SIZE = 32
EPOCHS=5 
AUTOTUNE = tf.data.AUTOTUNE

TFRECORD_FILE_TRAIN = # YOUR_PATH
TFRECORD_FILE_VALIDATION= # YOUR_PATH
TFRECORD_FILE_TEST = # YOUR_PATH
SIZE_DATASET_TRAIN= 40000
STEPS_PER_EPOCH = (SIZE_DATASET_TRAIN*2)//BATCH_SIZE
VALIDATION_STEP = 10000//BATCH_SIZE
model = get_model_data_augmentation_CPU() # Get the Keras model
data_augmentation = tf.keras.Sequential([
  layers.RandomFlip("horizontal_and_vertical"),
  layers.RandomRotation(0.2),
  layers.RandomZoom(0.2)
])
dataset_train = tf.data.TFRecordDataset(TFRECORD_FILE_TRAIN, num_parallel_reads=4)
dataset_train = dataset_train.repeat(2*EPOCHS)
dataset_train = dataset_train.map(parse_image_function, num_parallel_calls=8) # decode image from Tfrecord
dataset_train = dataset_train.map(lambda x,y: (data_augmentation(x, training=True),y), num_parallel_calls=8)
dataset_train = dataset_train.batch(BATCH_SIZE)
# Comment or uncomment following line to see differences 
dataset_train = dataset_train.prefetch(1) # I tried also with other buffer_size

# Training of the network
start_time = time.time() # Measure the start time of the training
history = model.fit(
    dataset_train,
    epochs=EPOCHS,
    steps_per_epoch=STEPS_PER_EPOCH,
)
end_time = time.time() 
training_time = end_time-start_time
print(f"Execution time: {(end_time-start_time)} sec")
```


### Relevant log output

_No response_</details>