Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations
@bkocis Could you please provide the complete standalone code to replicate this issue?
Thank you! I would need to rewrite a big chunk of the example code to remove very specific implementation code. 
I wanted to ask a more general question regarding the memory usage increase that increases over the training epochs. 

Searching the net, I found that this is most likely due to how tf handles its graph and stores the gradients. 

Specifically, the model.fit method could be the cause of this increase on memory usage, due to how it handles the mode, the data, and stores parameters. 

It appears that during training, one can not free up memory by deleting the model or clearing the session, or similar. 

I found this as the only post that was highly ranked as a good answer: 
https://github.com/tensorflow/tensorflow/issues/36465#issuecomment-582749350 

Other references to this are: 
https://github.com/keras-team/keras/issues/15887
https://github.com/tensorflow/tensorflow/issues/35030

Is there any simple way to free up memory while training (tf 2.11.0)?  Things like this don't work: 
```python
from keras import backend as K
K.clear_session()
# or 
 tf.keras.backend.clear_session()
``` Hello, @bkocis! If you want to allocate specific memory or increase the memory growth then, please have a look at this [page](https://www.tensorflow.org/guide/gpu#limiting_gpu_memory_growth) which clearly explains about how to limit the gpu memory growth. There are mainly two options;
Firstly, to turn on memory growth by calling [tf.config.experimental.set_memory_growth](https://www.tensorflow.org/api_docs/python/tf/config/experimental/set_memory_growth), which attempts to allocate only as much GPU memory as needed for the runtime allocations 
```
gpus = tf.config.list_physical_devices('GPU')
if gpus:
  try:
    # Currently, memory growth needs to be the same across GPUs
    for gpu in gpus:
      tf.config.experimental.set_memory_growth(gpu, True)
    logical_gpus = tf.config.list_logical_devices('GPU')
    print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
  except RuntimeError as e:
    # Memory growth must be set before GPUs have been initialized
    print(e)
```

And secondly, to configure a virtual GPU device with [tf.config.set_logical_device_configuration](https://www.tensorflow.org/api_docs/python/tf/config/set_logical_device_configuration) and set a hard limit on the total memory to allocate on the GPU
```
gpus = tf.config.list_physical_devices('GPU')
if gpus:
  # Restrict TensorFlow to only allocate 1GB of memory on the first GPU
  try:
    tf.config.set_logical_device_configuration(
        gpus[0],
        [tf.config.LogicalDeviceConfiguration(memory_limit=1024)])
    logical_gpus = tf.config.list_logical_devices('GPU')
    print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
  except RuntimeError as e:
    # Virtual devices must be set before GPUs have been initialized
    print(e)
```
Please let me know if it helps?
Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. @sushreebarsa 
I implemented this - it got me further. But this alone did not solve my issues. 
I can't confirm yet, but what helped was to remove validation dataset from the training (model.fit). Do you know why that might be the case?
 Any updates on this?  Hi, Along with the linked issues there is one more issue which is open since long. if you find this issue is related to your issue, could you please close this issue and track the progress in the linked issue. https://github.com/tensorflow/tensorflow/issues/58676 The above linked issue is closed with a fix, please refer and let us know if that fix works for you. Hi @sachinprasadhs - as far as I can understand, this fix is in 2.15 version. This is not yet on pip - any ideas when this will come out?  We initially release 2.15rc0, which should be ideally released within a week and after few more weeks final release will happen.
Till then you can use tf-nightly which will include the above changes.  Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61816">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61816">No</a>

### Issue type

Performance

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

binary

### TensorFlow version

2.11.0

### Custom code

Yes

### OS platform and distribution

Linux Ubuntu 20.04

### Mobile device

_No response_

### Python version

3.10

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

I have an issue with memory consumption during training. After each epoch more memory is used. In case of bigger dataset, the training crashed mid training. 
I use loading the training data from generator, where I sample the batch using pandas sample method. 
One of the warnings from tf is: 

`W tensorflow/core/framework/dataset.cc:769] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.`

I am a bit confused, whether using the generator and sampling from a pandas dataframe can cause memory accumulation during the training?



### Standalone code to reproduce the issue

```shell
def generate_samples(epochs: int, steps_per_epoch: int, token_y_df: pd.DataFrame, num_fields: int, batch_size: int = 32, max_len: int = 1024) -> tuple[np.array, np.array]:
        pad_value_y = get_padding_value_y(num_fields)
        while True:
            batch_df = token_y_df.sample(batch_size, replace=True)
            _x, _y = get_fixed_batch(batch_df, max_len, pad_value_y=pad_value_y,
                                     use_additional_features=use_additional_features)
            yield _x, _y

model.fit(generate_samples, ....)


The `get_padding` and the `get_fixed_batch` are longer functions, that pad and create the X and y np.arrays from the sampled pandas dataframe.
```


### Relevant log output

```shell
W tensorflow/core/framework/dataset.cc:769] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.
```
