Memory leak in model fit with dataset from generator
@arianmaghsoudnia,
Thank you for reporting the issue.

Could you please take a look at this relevant issue threads https://github.com/tensorflow/tensorflow/issues/58606, https://github.com/tensorflow/tensorflow/issues/57690, https://github.com/tensorflow/tensorflow/issues/56624.

Also I would suggest you to have look at this article [tf.data.Dataset generators with parallelization](https://medium.com/@acordier/tf-data-dataset-generators-with-parallelization-the-easy-way-b5c5f7d2a18) which may help you to solve your memory related issue. Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60659">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60659">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Performance

### Have you reproduced the bug with TF nightly?

No

### Source

binary

### Tensorflow Version

2.12.0

### Custom Code

Yes

### OS Platform and Distribution

Linux Ubuntu 22.04

### Mobile device

_No response_

### Python version

3.10, 3.11

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

None

### GPU model and memory

None

### Current Behaviour?

When creating a dataset from the generator, using Tensorflow V2.12.0 with Python 3.11, some memory is not released after training on each batch, leading to a linear increase in memory usage during the model fit step.

### Standalone code to reproduce the issue

```shell
This is the code I used to generate random data on the fly to be used by the generator, and a simple model to fit this data on. 


batch_size =32
train_datapoints_count = 1000000
val_datapoints_count = 200000

def generate_random_dataset(input_size):
    for _ in range(input_size):
        x = np.random.rand(batch_size, 1, 10).astype(np.float32)
        y = np.random.rand(batch_size, 1, 1).astype(np.float32)
        yield x, y

dataset_train = tf.data.Dataset.from_generator(
    generator=generate_random_dataset,
    args=[train_datapoints_count],
    name="random_ds_train",
    output_signature=(
        tf.TensorSpec(shape=(None, 1, 10), dtype=np.float32),
        tf.TensorSpec(shape=(None, 1, 1), dtype=np.float32),
    ),
)

dataset_val = tf.data.Dataset.from_generator(
    generator=generate_random_dataset,
    args = [val_datapoints_count],
    name="random_ds_val",
    output_signature=(
        tf.TensorSpec(shape=(None, 1, 10), dtype=np.float32),
        tf.TensorSpec(shape=(None, 1, 1), dtype=np.float32),
    ),
)

model = tf.keras.Sequential(
    [
        tf.keras.layers.Flatten(),
        tf.keras.layers.Dense(units=batch_size, activation="relu"),
        tf.keras.layers.Dense(units=batch_size, activation="relu"),
        tf.keras.layers.Dense(units=1),
    ]
)

model.compile(
    loss=tf.losses.MeanSquaredError(),
    optimizer=tf.optimizers.Adam(),
    metrics=[tf.metrics.MeanAbsoluteError(), tf.metrics.MeanSquaredError()],
)
model.fit(
    dataset_train,
    epochs=2,
    validation_data=dataset_val,
)
```

I then ran the code in a container and logged the memory usage.
```
FROM python:3.10 
#also 3.11

WORKDIR /app

RUN pip install tensorflow==2.12.0
COPY train.py .

ENTRYPOINT ["python", "train.py"]
```
```


### Relevant log output

```shell
With Python 3.11, I see an increment in memory usage over time, while it remains the same for Python 3.10. I am allocating the exact same resources to both of the containers.
The columns are respectively: Seconds into the fitting step, memory usage tf_2.12_python_3.10, memory usage tf_2.12_python_3.11. 

5	254.9	288.9
10	254.9	291
15	254.9	293.1
20	254.9	294.8
25	254.9	296.4
30	254.9	298.5
35	254.9	300.3
40	254.9	301.8
45	254.9	304
50	254.9	306.2
55	254.9	308.3
60	254.9	310.6
65	254.9	312.5
70	254.9	314.8
75	254.9	316.5
80	254.9	317.9
85	254.6	319.5
90	254.2	320.4
95	253.8	321.5
100	253	323.1
```
</details>