Significant difference in RSS memory usage between TF1 and TF2
@SuryanarayanaY, please take a look at this ASAP.  ![image](https://github.com/tensorflow/tensorflow/assets/15092622/4262894b-d06e-4aba-be23-75202bd0b8d2)

If it helps, running it through `memray` and generating a flamegraph with `--leaks`, we see the above as having ~18 GB memory allocated from 27k allocations. So, likely that something is up with those. Hi @bergentruckung ,

You need to call `tf.keras.backend.clear_session()` in the loop to remove the global states generated by keras. Could you please add this at the beginning of epoch and let us know.

Thanks! Hi @SuryanarayanaY,

I tried that out, but it doesn't seem to work. Hi @bergentruckung ,

I did replicated the behaviour with Keras3-nightly also.Attached [gist](https://colab.research.google.com/drive/1ewHE0e6PtygZYwyZiqbqNtVnEJBYB23G?resourcekey=0-UAj6TMTN-39OBfWYP3QbCw) for reference. Not sure whether this needs to be addressed in Keras repo or here itself. Could you please open an issue at [Keras](https://github.com/keras-team/keras/issues) repo also. Hi @SuryanarayanaY,

I think your suggested workaround may work for newer TF that has changes from https://github.com/tensorflow/tensorflow/pull/62154 pulled in. This wasn't cherry-picked onto 2.13 branch.

Could you backport that PR to 2.13? @bergentruckung Could you please upgrade to the latest if possible as backported changes in 2.13 may introduce new bugs or regressions. It's unlikely for older versions to receive any bug fixes except when we have security patches. 
Thank you! We're building from source. We were able to build 2.14, but that has cuBLAS open issue that's problematic for us (the fix was to go back to 2.13 and that's what we did). We couldn't find a proper closure of dependencies for 2.15, and that's why we were trying previous versions. @bergentruckung Could you please let us know if you have created another issue in Keras repo as suggested [here](https://github.com/tensorflow/tensorflow/issues/62839#issuecomment-1916579409)?
Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62839">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62839">No</a>

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

source

### TensorFlow version

TF 2.13.1

### Custom code

Yes

### OS platform and distribution

Redhat Enterprise Linux 8.9

### Mobile device

_No response_

### Python version

3.11.4

### Bazel version

5.4.0

### GCC/compiler version

10.4

### CUDA/cuDNN version

CUDA 12.2, cuDNN 8.9.5

### GPU model and memory

A100 80GB

### Current behavior?

When running the same keras workload on TF1 vs. TF2, I'm seeing a significant increase in memory utilization per epoch. This happens when using both CPUs/GPUs. The utilization for an epoch climbs up significantly after every epoch. See below for TF1 vs. TF2:

```
# For TF2

Memory usage after epoch 0 [mem_usage =  3.41 GB] 
Memory usage after epoch 1 [mem_usage =  3.88 GB] 
Memory usage after epoch 2 [mem_usage =  3.88 GB] 
Memory usage after epoch 3 [mem_usage =  4.32 GB] 
Memory usage after epoch 4 [mem_usage =  4.81 GB] 
Memory usage after epoch 5 [mem_usage =  5.26 GB] 
Memory usage after epoch 6 [mem_usage =  5.70 GB] 
Memory usage after epoch 7 [mem_usage =  6.14 GB] 
Memory usage after epoch 8 [mem_usage =  6.70 GB] 
Memory usage after epoch 9 [mem_usage =  7.15 GB] 
Memory usage after epoch 10 [mem_usage =  7.36 GB]
Memory usage after epoch 11 [mem_usage =  7.36 GB]
Memory usage after epoch 12 [mem_usage =  7.36 GB]
Memory usage after epoch 13 [mem_usage =  7.37 GB]
Memory usage after epoch 14 [mem_usage =  7.37 GB]
Memory usage after epoch 15 [mem_usage =  7.37 GB]
Memory usage after epoch 16 [mem_usage =  7.37 GB]
Memory usage after epoch 17 [mem_usage =  7.59 GB]
Memory usage after epoch 18 [mem_usage =  7.81 GB]
Memory usage after epoch 19 [mem_usage =  7.81 GB]

# For TF1

Memory usage after epoch 0 [mem_usage =  5.13 GB] 
Memory usage after epoch 1 [mem_usage =  5.14 GB] 
Memory usage after epoch 2 [mem_usage =  5.14 GB] 
Memory usage after epoch 3 [mem_usage =  5.15 GB] 
Memory usage after epoch 4 [mem_usage =  5.15 GB] 
Memory usage after epoch 5 [mem_usage =  5.15 GB] 
Memory usage after epoch 6 [mem_usage =  5.15 GB] 
Memory usage after epoch 7 [mem_usage =  5.15 GB] 
Memory usage after epoch 8 [mem_usage =  5.15 GB] 
Memory usage after epoch 9 [mem_usage =  5.15 GB] 
Memory usage after epoch 10 [mem_usage =  5.15 GB]
Memory usage after epoch 11 [mem_usage =  5.15 GB]
Memory usage after epoch 12 [mem_usage =  5.15 GB]
Memory usage after epoch 13 [mem_usage =  5.15 GB]
Memory usage after epoch 14 [mem_usage =  5.15 GB]
Memory usage after epoch 15 [mem_usage =  5.15 GB]
Memory usage after epoch 16 [mem_usage =  5.15 GB]
Memory usage after epoch 17 [mem_usage =  5.15 GB]
Memory usage after epoch 18 [mem_usage =  5.15 GB]
Memory usage after epoch 19 [mem_usage =  5.15 GB]
```

### Standalone code to reproduce the issue

```shell
import tensorflow as tf
import psutil
import time
import os

def mem_usage_str():
    process = psutil.Process(os.getpid())
    gb =  process.memory_info().rss / (1024.**3)
    return ' [mem_usage = {:5.2f} GB]'.format(gb)

if int(tf.__version__.split('.')[0]) < 2:
    """
    Patch to fix TF/numpy1.20 compatibility issue
    """
    from   tensorflow.math          import reduce_prod
    from   tensorflow.python.ops    import array_ops

    def _constant_if_small(value, shape, dtype, name):
        try:
            if reduce_prod(shape) < 1000:  # monkey patch
                return array_ops.constant(value, shape=shape, dtype=dtype,
                                          name=name)
        except TypeError:
            # Happens when shape is a Tensor, list with Tensor elements, etc.
            pass
        return None

    array_ops._constant_if_small = _constant_if_small
    """
    End of patch
    """

def build_model():
    inputs = [tf.keras.layers.Input(shape=(300, 6), name='input_layer')]
    current_layer = inputs[0]

    current_layer = tf.keras.layers.LSTM(
        50,
        dropout=0.1,
        recurrent_dropout=0.1,
        return_sequences=False,
        name='lstm',
    )(current_layer)

    current_layer = tf.keras.layers.Dense(1)(current_layer)
    optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)

    model = tf.keras.models.Model(inputs=inputs, outputs=current_layer)
    model.compile(loss='mse', optimizer=optimizer)

    return model

def run(model, X, y, n_epochs):
    tot_time = 0.

    print(f'Memory usage before training' + mem_usage_str())
    for i in range(n_epochs):
        start = time.time()
        model.fit(X, y, epochs=1, batch_size=4096, verbose=0)
        tot_time += time.time() - start
        print(f'Memory usage after epoch {i}' + mem_usage_str())

    print(f'Avg. time = {tot_time / n_epochs} seconds')

def run_example(p, n_epochs):
    import numpy as np

    model = build_model()
    X = np.random.randn(2 ** p, 300, 6)
    y = np.random.randn(2 ** p)

    run(model, X, y, n_epochs)

def main():
    run_example(
        16, # 2 ** 16 samples
        20, # 10 epochs
    )


# ------------------------------------------------------------------------------

if __name__ == "__main__":
    main()
```


### Relevant log output

_No response_