Memory leak in forward pass (e.g., of ResNet50 model) with TensorFlow 2.12.0 and Python 3.11
Have you tried using ResNet50V2 I got a similar issue when I ran your code but when I tried it with ResNet50V2 I did not get a memory leak

```
import numpy as np
import psutil
import tensorflow as tf


model = tf.keras.applications.ResNet50V2()  # VGG19 seems to not leak.

# tf.config.threading.set_inter_op_parallelism_threads(0) and tf.config.threading.set_intra_op_parallelism_threads(0) do not help.

inp = (np.random.rand(1, 224, 224, 3) * 255).astype('uint8')


for run in range(1, 9999999):
    model(inp)
    memory_usage_in_MiB = psutil.Process().memory_info().rss / (1024 * 1024)
    print(
        f'Memory usage after {run} run(s) (in MiB): {memory_usage_in_MiB:.3f}', flush=True)
```


```
Memory usage after 1753 run(s) (in MiB): 348.219
Memory usage after 1754 run(s) (in MiB): 348.875
Memory usage after 1755 run(s) (in MiB): 337.938
Memory usage after 1756 run(s) (in MiB): 336.734
Memory usage after 1757 run(s) (in MiB): 339.516
Memory usage after 1758 run(s) (in MiB): 345.094
Memory usage after 1759 run(s) (in MiB): 340.312
Memory usage after 1760 run(s) (in MiB): 339.547
Memory usage after 1761 run(s) (in MiB): 340.281
Memory usage after 1762 run(s) (in MiB): 341.266
Memory usage after 1763 run(s) (in MiB): 339.000
Memory usage after 1764 run(s) (in MiB): 340.906
Memory usage after 1765 run(s) (in MiB): 340.469
Memory usage after 1766 run(s) (in MiB): 340.391
Memory usage after 1767 run(s) (in MiB): 341.953
Memory usage after 1768 run(s) (in MiB): 339.672
Memory usage after 1769 run(s) (in MiB): 342.922
Memory usage after 1770 run(s) (in MiB): 346.969
Memory usage after 1771 run(s) (in MiB): 318.844
``` I just tested with `ResNet50V2`, and I get the memory leak there too: https://gist.github.com/Dobiasd/d2cabf576e1dd518610a2387b7772ecd

Did you use the provided `Dockerfile` to make sure you have the same versions of Python and TensorFlow? yeah my version of TensorFlow was wrong I was using the most up to date version from source, with the docker file I get the memory leak with ResNet50V2 as well Hi!

Thank you for sharing! Could you help us out by running a bisect using your dockerfile? 

You can use the nightly releases to pinpoint the version where it first appears. That alone should narrow it down a lot. But if you can identify the commit too that would be great! @Dobiasd Could you please refer to the comment above and update on the same?
Thank you! I just tried to ([available versions](https://gist.github.com/Dobiasd/fc4f72a8576e7524bdc4ccf3871fdf2c) of `tf-nightly` shown by `pip`, `2.12.0.dev20230105` being the oldest one), but some versions throw an error right from the beginning. Here are my results (focusing on the transitions where something changes):
- `2.12.0.dev20230105`: [error](https://gist.github.com/Dobiasd/706db6cf6bd1477181d119c8d2cecbfe)
- `2.12.0.dev20230127`: [error](https://gist.github.com/Dobiasd/3472e2cdb5ece498aa9519f55a2bddc8)
- `2.12.0.dev20230201`: memleak
- `2.12.0.dev20230203`: memleak
- `2.13.0.dev20230206`: [error](https://gist.github.com/Dobiasd/11b5ed8689cb3dacae9d155c2992f7d0)
- `2.13.0.dev20230405`: memleak

So I did not find the version that introduced the memleak, because it looks like it was introduced before the earliest version, which can be tested that way.

For further attempts, feel free to use the code, I provided, on your own. :wink: I have replicated the issue on Colab also with tf-nightly version and with python 3.9.16.

Please refer the attached [gist](https://colab.research.google.com/gist/SuryanarayanaY/21ab1b8d473948390ce22adb74a9f852/60131.ipynb) for details.
 Yea I raised this issue a long time ago. `model.predict` has memory leak in more recent version of Tensorflow.
It still open. It stopped us from upgrading TF on our production backend.
https://github.com/tensorflow/tensorflow/issues/58676 cc: @sampathweb , @sagunb  See also #62203
 Any news on this ? The issue is still present in tf-nighly and not specific to keras

`Dockerfile`
```Dockerfile
FROM python:3.10.11
#FROM python:3.11.2

RUN pip install --no-cache-dir tf-nightly-cpu psutil==5.9.4

# Disable the Docker cache from this stage on, see https://stackoverflow.com/a/58801213/1866775
ADD "https://www.random.org/cgi-bin/randbyte?nbytes=10&format=h" skipcache

ADD ./memleak.py /
RUN python /memleak.py
```

`memleak.py`:
```python
import os, psutil
process = psutil.Process()
print(process.memory_info().rss / 1000000)  # Ram usage in MB

import tensorflow as tf
print(tf.__version__)

while True:
    print(process.memory_info().rss / 1000000, flush=True)
    x = tf.random.normal(shape=(1,))  # Causing growing RAM
``` I ran `memray` against the same python script in 3.10 and 3.11 

```python
import tensorflow as tf
for _ in range(10000):
    for _ in range(100):
        x = tf.random.normal(shape=(1,))  # Causing growing RAM
```

The memory flamegraphs look quite different
py310 https://drive.google.com/file/d/1zLq-k2jRJqnc9l_ZRwWuN_VU6SL8-2gc/view?usp=drive_link
py311 https://drive.google.com/file/d/1USJ9f5rkoza_wAnTG5iig1Bf2zuF03Da/view?usp=drive_link Hi @Dobiasd ,

This got fixed. I have checked the code with latest Tf versions TF2.15( with python-3.11v) and tf-nightly(with python-3.10) and no memory leak observed now. Please refer attached [gist](https://colab.sandbox.google.com/gist/SuryanarayanaY/a198b94ac79152b37484a48835a3d7ed/60131_tf-nightly-py-3-11.ipynb#scrollTo=vp62uDa0gKS7).

Could you please verify and confirm. Thanks! Just tested changing `tensorflow==2.12.0` to `tensorflow==2.15.0.post1` (the latest TF 2.15) in [my original example](https://github.com/tensorflow/tensorflow/issues/60131#issue-1642338990), and the memory leak still exists: https://gist.github.com/Dobiasd/e1a5bd31378a92b690388bb62dffbcee

Can you reproduce it? (maybe try to do it not in a Jupyter Notebook, but with Docker as in my example) @SuryanarayanaY In your gist you're not actually running against a python3.11 kernel, simply print `sys.version` to check...
It seems that there's no easy way to install python3.11 in colab except for something like https://colab.research.google.com/drive/13C50iNZRjMRyepe_3xAtMKUQdIkKK0kS For an additional data point I ran the above gist with python 3.11.8 and tensorflow 2.16.1 with the following results.
Memory usage after 100 run(s) (in MiB): 867.309
Memory usage after 200 run(s) (in MiB): 893.266
Memory usage after 300 run(s) (in MiB): 872.895
Memory usage after 400 run(s) (in MiB): 898.473
Memory usage after 500 run(s) (in MiB): 884.656
Memory usage after 600 run(s) (in MiB): 900.805
Memory usage after 700 run(s) (in MiB): 905.102
Memory usage after 800 run(s) (in MiB): 915.559
Memory usage after 900 run(s) (in MiB): 947.211
Memory usage after 1000 run(s) (in MiB): 942.445
Memory usage after 1100 run(s) (in MiB): 967.773
Memory usage after 1200 run(s) (in MiB): 993.973
Memory usage after 1300 run(s) (in MiB): 983.297
Memory usage after 1400 run(s) (in MiB): 977.977
Memory usage after 1500 run(s) (in MiB): 988.398
Memory usage after 1600 run(s) (in MiB): 998.324
Memory usage after 1700 run(s) (in MiB): 1033.254
Memory usage after 1800 run(s) (in MiB): 1028.336
Memory usage after 1900 run(s) (in MiB): 1047.902
Memory usage after 2000 run(s) (in MiB): 1039.895

I also tested python 3.11.8 and tensorflow 2.13.1 & tensorflow 2.15.1 both versions also have the memory leak. For others running into this issue I tested predict_on_batch approach instead of using the __call__ method and the memory leak is not present.

```
import numpy as np
import psutil
import tensorflow as tf
print(tf.__version__)

model = tf.keras.applications.ResNet50()  # VGG19 seems to not leak.

inp = (np.random.rand(1, 224, 224, 3) * 255).astype('uint8')

for run in range(1, 5000):
    model.predict_on_batch(inp)
    memory_usage_in_MiB = psutil.Process().memory_info().rss / (1024 * 1024)
    if run%100 == 0:
        print(f'Memory usage after {run} run(s) (in MiB): {memory_usage_in_MiB:.3f}', flush=True)
```

Memory usage after 100 run(s) (in MiB): 962.145
Memory usage after 200 run(s) (in MiB): 981.012
Memory usage after 300 run(s) (in MiB): 981.910
Memory usage after 400 run(s) (in MiB): 981.195
Memory usage after 500 run(s) (in MiB): 976.898
Memory usage after 600 run(s) (in MiB): 980.027
Memory usage after 700 run(s) (in MiB): 971.473
Memory usage after 800 run(s) (in MiB): 973.348
Memory usage after 900 run(s) (in MiB): 977.871
Memory usage after 1000 run(s) (in MiB): 981.832
Memory usage after 1100 run(s) (in MiB): 982.309
Memory usage after 1200 run(s) (in MiB): 982.062
Memory usage after 1300 run(s) (in MiB): 984.105
Memory usage after 1400 run(s) (in MiB): 798.184
Memory usage after 1500 run(s) (in MiB): 800.445
Memory usage after 1600 run(s) (in MiB): 799.742
Memory usage after 1700 run(s) (in MiB): 801.027
Memory usage after 1800 run(s) (in MiB): 802.859
Memory usage after 1900 run(s) (in MiB): 801.484
Memory usage after 2000 run(s) (in MiB): 802.805

This test was run with python 3.11.8 and tensorflow 2.13.1 @Dobiasd,
Could you please confirm, whether this is the same issue where we are tracking in the keras-team/Keras repo.
https://github.com/keras-team/keras/issues/19500

Please correct me if I understand the right. Thank you Yes, the two issues are similar.

The difference is, that in this one here, opened in 2023-03, the memory usage leaked slowly but steadily.

In the other one, opened in 2024-04, the memory grows very quickly immediately and then stays somewhat constant.
The following minimal example reproduces the memory leak I ran into. (No GPU, just CPU.)

`memleak.py`:

```python3
import numpy as np
import psutil
import tensorflow as tf

model = tf.keras.applications.ResNet50()  # VGG19 seems to not leak.

# tf.config.threading.set_inter_op_parallelism_threads(0) and tf.config.threading.set_intra_op_parallelism_threads(0) do not help.

inp = (np.random.rand(1, 224, 224, 3) * 255).astype('uint8')

for run in range(1, 9999999):
    model(inp)
    memory_usage_in_MiB = psutil.Process().memory_info().rss / (1024 * 1024)
    print(f'Memory usage after {run} run(s) (in MiB): {memory_usage_in_MiB:.3f}', flush=True)
```

`Dockerfile`:
```Dockerfile
FROM python:3.11.2

RUN pip install --no-cache-dir tensorflow==2.12.0 psutil==5.9.4

# Disable the Docker cache from this stage on, see https://stackoverflow.com/a/58801213/1866775
ADD "https://www.random.org/cgi-bin/randbyte?nbytes=10&format=h" skipcache

ADD ./memleak.py /
RUN python /memleak.py
```

[Output](https://gist.github.com/Dobiasd/ba800b40e117aa13d97deb44761888f6) (`docker build --rm .`):
```
Memory usage after 1 run(s) (in MiB): 604.324
Memory usage after 2 run(s) (in MiB): 606.906
Memory usage after 3 run(s) (in MiB): 606.906
Memory usage after 4 run(s) (in MiB): 606.906
Memory usage after 5 run(s) (in MiB): 606.906
Memory usage after 6 run(s) (in MiB): 607.164
Memory usage after 7 run(s) (in MiB): 607.164
Memory usage after 8 run(s) (in MiB): 607.164
Memory usage after 9 run(s) (in MiB): 607.164
Memory usage after 10 run(s) (in MiB): 607.164
Memory usage after 11 run(s) (in MiB): 607.422
Memory usage after 12 run(s) (in MiB): 607.422
[...]
Memory usage after 498 run(s) (in MiB): 626.242
Memory usage after 499 run(s) (in MiB): 626.242
Memory usage after 500 run(s) (in MiB): 626.242
Memory usage after 501 run(s) (in MiB): 626.500
Memory usage after 502 run(s) (in MiB): 626.500
[...]
[...]
Memory usage after 1996 run(s) (in MiB): 683.477
Memory usage after 1997 run(s) (in MiB): 683.734
Memory usage after 1998 run(s) (in MiB): 683.734
Memory usage after 1999 run(s) (in MiB): 683.734
Memory usage after 2000 run(s) (in MiB): 683.734
Memory usage after 2001 run(s) (in MiB): 683.734
[...]
Memory usage after 9996 run(s) (in MiB): 960.258
Memory usage after 9997 run(s) (in MiB): 960.508
Memory usage after 9998 run(s) (in MiB): 960.508
Memory usage after 9999 run(s) (in MiB): 960.508
Memory usage after 10000 run(s) (in MiB): 960.508
Memory usage after 10001 run(s) (in MiB): 960.508
[...]
Memory usage after 24997 run(s) (in MiB): 1547.840
Memory usage after 24998 run(s) (in MiB): 1547.840
Memory usage after 24999 run(s) (in MiB): 1534.230
Memory usage after 25000 run(s) (in MiB): 1532.348
Memory usage after 25001 run(s) (in MiB): 1533.441
Memory usage after 25002 run(s) (in MiB): 1544.711
[...]
```

When using the same TensorFlow version (`2.12.0`) but with Python `3.10.10` (instead of `3.11.2`), the memory usage does not grow.