Memory leak when using tf.Model and tf.Model.fit() in a loop. clear_session() does not help
@alessiomora Could you please confirm if all the dependencies are shared? I tried to replicate the issue reported [here](https://colab.research.google.com/gist/sushreebarsa/b8efa42dcc80605890f82b56360f18ae/61791.ipynb), please let me know what I am missing here. Thank you! @sushreebarsa it's hard to reproduce this on colab. Anyway, I simplified the code and you can run the following code on colab. Also, it would be a lot easier with a GPU. Also, the printed results are reliable if this is the only process running (in my machine I can be sure about that, I am not in colab).

Thank you in advance!
```
!pip install tensorflow==2.13
!pip install psutil==5.9.5

import tensorflow as tf # same issue with tf-nightly = "2.15.0.dev20230904"
import time
import gc
import psutil # psutil == "5.9.5"
import subprocess as sp

class MyModel(tf.keras.Model):

    def __init__(self):
        super().__init__()
        self.dense1 = tf.keras.layers.Dense(1000, activation=tf.nn.relu)
        self.dense2 = tf.keras.layers.Dense(10000, activation=tf.nn.relu)
        self.dense3 = tf.keras.layers.Dense(10000, activation=tf.nn.relu)
        self.dense4 = tf.keras.layers.Dense(1000, activation=tf.nn.softmax)

    def call(self, inputs):
        x = self.dense1(inputs)
        x = self.dense2(x)
        x = self.dense3(x)
        x = self.dense4(x)
        return x


if __name__ == '__main__':
    print(f"Starting..")
    memory_free_val_initial, memory_perc_initial = get_cpu_memory()
    print(f"[Memory monitoring] Free memory CPU {memory_free_val_initial} MB, {memory_perc_initial} %.")

    for r in range(0, 1000):
        model = MyModel()
        # ds = tf.data.Dataset.from_tensor_slices((tf.random.uniform((64*4, 1000)), tf.ones((64*4))))
        ds = (lambda: tf.data.Dataset.from_tensor_slices((tf.random.uniform((64 * 20, 1000)), tf.ones((64 * 20)))))
        model.compile(optimizer='sgd', loss=tf.keras.losses.SparseCategoricalCrossentropy())

        model.fit(ds().batch(64), verbose=0)
        model.evaluate(ds().batch(64), verbose=0)
        tf.keras.backend.clear_session()

        if r % 5 == 0:
            # print every 5 model.fit
            print(f"Round: {r}")
            memory_free_val, memory_perc = get_cpu_memory()
            print(f"[Memory monitoring] Free memory CPU {memory_free_val} MB, {memory_perc} %.")
            if r == 0:
                memory_free_first = memory_free_val

        del model
        gc.collect()
        del ds

    print(f"[Memory monitoring CPU] Memory usage increased by {memory_free_first - memory_free_val} MB, "
          "during the process.")

```

 Hi @alessiomora ,

I have checked the code snippet with different functions and listed the results below.

1.With  `psutil.virtual_memory()` and **CPU** runtime: 

Here I have tested for 100 epochs and I observed after 2nd round the memory is oscillating around a fixed value.Not changing much.[gist](https://colab.sandbox.google.com/gist/SuryanarayanaY/ec8e035019714802f9bbae59f84ddff8/61791_r1.ipynb). Not sure with 1000 epochs it will change much as it is time taking. Could we need to test for 1000 epochs? Please confirm.

2. With **GPU** runtime and using `tf.config.experimental.get_memory_info()`

I have checked with GPU runtime also using TF inbuilt function tf.config.experimental.get_memory_info and GPU memory stayed at constant fixed value. Here I have tested for 300 epochs.Please find the attached [gist](https://colab.sandbox.google.com/drive/1qYJimUa7UA6-lPK3NVWgenO-qNDj6cV-?resourcekey=0-jaV4kAHfXOyzlcpHXdonJg#scrollTo=BVQl3WdWxVsi) for reference.

If I consider GPU part I don't observe Memory leak.

Could you please check and come back with your comments.

Thanks! @SuryanarayanaY thank you. 
For the constant value of GPU memory usage: as my first attached code and results were showing, also in my case GPU memory does not increase significantly.
For CPU memory usage, can you reproduce the code in a Linux/Ubuntu OS?
In the results I provided,  in 1000 rounds there is an increase of 6767 MB (also consider that the dataset and the model used in this exemplary code are just toys! The problem is way worse with a more complex setting).
I would like to know if you are not experiencing the issue with 1000 rounds, and also if you are able to reproduce the issue outside colab.
Thank you so much! Hi @alessiomora ,

This [PR](https://github.com/tensorflow/tensorflow/pull/62154) might addresses the issue now. Could you please check the behaviour with tf-nightly ? This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61791">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61791">No</a>

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes (tf-nightly = "2.15.0.dev20230904")

### Source

source

### TensorFlow version

2.13, 2.12, 2.11

### Custom code

Yes

### OS platform and distribution

Ubuntu 22.04.1 LTS (GNU/Linux 5.16.10 x86_64)

### Mobile device

_No response_

### Python version

3.10.0, 3.9.0

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

cuDNN 8600

### GPU model and memory

NVIDIA RTX A5000, 24GB

### Current behavior?

Memory usage steadily increases when using tf.keras.Model and tf.keras.Model.fit() in a loop, and leads to Out Of Memory exception saturating the memory eventually. clear_session() does not help. The same code with TF version == 2.9.2 has an almost constant memory usage instead, and works as expected.
I've also opened [this issue](https://github.com/keras-team/tf-keras/issues/286) on Keras's GitHub, months ago, with no solutions from the Keras team.

### Standalone code to reproduce the issue

```shell
import tensorflow as tf # same issue with tf-nightly = "2.15.0.dev20230904"
import time
import gc
import psutil # psutil == "5.9.5"
import subprocess as sp

gpus = tf.config.experimental.list_physical_devices('GPU')
for gpu in gpus:
    tf.config.experimental.set_memory_growth(
        device=gpu, enable=True
    )


def get_cpu_memory():
    memory_info = psutil.virtual_memory()
    # you can have the percentage of used RAM
    memory_percent = 100.0 - memory_info.percent
    memory_free_values = memory_info.available / (1024 * 1024)  # in MB
    # you can calculate percentage of available memory
    return memory_free_values, memory_percent


def get_gpu_memory():
    command = "nvidia-smi --query-gpu=memory.free --format=csv"
    memory_free_info = sp.check_output(command.split()).decode('ascii').split('\n')[:-1][1:]
    memory_free_values = [int(x.split()[0]) for i, x in enumerate(memory_free_info)][0]
    memory_percent = (memory_free_values / 24564) * 100  # my gpu has 24564 MB of memory
    return memory_free_values, memory_percent


class MyModel(tf.keras.Model):

    def __init__(self):
        super().__init__()
        self.dense1 = tf.keras.layers.Dense(1000, activation=tf.nn.relu)
        self.dense2 = tf.keras.layers.Dense(10000, activation=tf.nn.relu)
        self.dense3 = tf.keras.layers.Dense(10000, activation=tf.nn.relu)
        self.dense4 = tf.keras.layers.Dense(1000, activation=tf.nn.softmax)

    def call(self, inputs):
        x = self.dense1(inputs)
        x = self.dense2(x)
        x = self.dense3(x)
        x = self.dense4(x)
        return x


if __name__ == '__main__':
    print(f"Starting..")
    memory_free_val_initial, memory_perc_initial = get_cpu_memory()
    print(f"[Memory monitoring] Free memory CPU {memory_free_val_initial} MB, {memory_perc_initial} %.")
    memory_free_val_initial_gpu, memory_perc_initial_gpu = get_gpu_memory()
    print(f"[Memory monitoring] Free memory GPU {memory_free_val_initial_gpu} MB, {memory_perc_initial_gpu} %.")

    for r in range(0, 1000):
        model = MyModel()
        # ds = tf.data.Dataset.from_tensor_slices((tf.random.uniform((64*4, 1000)), tf.ones((64*4))))
        ds = (lambda: tf.data.Dataset.from_tensor_slices((tf.random.uniform((64 * 20, 1000)), tf.ones((64 * 20)))))
        model.compile(optimizer='sgd', loss=tf.keras.losses.SparseCategoricalCrossentropy())

        model.fit(ds().batch(64), verbose=0)
        model.evaluate(ds().batch(64), verbose=0)
        tf.keras.backend.clear_session()

        if r % 5 == 0:
            # print every 5 model.fit
            print(f"Round: {r}")
            memory_free_val, memory_perc = get_cpu_memory()
            print(f"[Memory monitoring] Free memory CPU {memory_free_val} MB, {memory_perc} %.")
            memory_free_val_gpu, memory_perc_gpu = get_gpu_memory()
            print(f"[Memory monitoring] Free memory GPU {memory_free_val_gpu} MB, {memory_perc_gpu} %.")
            if r == 0:
                memory_free_first = memory_free_val
                memory_free_first_gpu = memory_free_val_gpu
            # time.sleep(2)

        del model
        gc.collect()
        del ds

    print(f"[Memory monitoring CPU] Memory usage increased by {memory_free_first - memory_free_val} MB, "
          "during the process.")
    print(f"[Memory monitoring GPU] Memory usage increased by {memory_free_first_gpu - memory_free_val_gpu} MB, "
          "during the process.")
```


### Relevant log output

```shell
Round: 0
[Memory monitoring] Free memory CPU 56633.48046875 MB, 88.5 %.
[Memory monitoring] Free memory GPU 21180 MB, 86.2237420615535 %.


Round: 995
[Memory monitoring] Free memory CPU 49866.3046875 MB, 77.9 %.
[Memory monitoring] Free memory GPU 21156 MB, 86.12603810454324 %.

[Memory monitoring CPU] Memory usage increased by 6767.17578125 MB, during the process.
[Memory monitoring GPU] Memory usage increased by 24 MB, during the process.
```
