Functions that limit video memory do not work
Hi, 

Thanks for reporting the issue.

In order to find the root cause of the issue, could you please help us by providing sample reproducible code. > Hi,
> 
> Thanks for reporting the issue.
> 
> In order to find the root cause of the issue, could you please help us by providing sample reproducible code.

**main.py**
```c
import os
import time
from importlib import import_module
from pathlib import Path


def run(model_name):
    for i in range(1, 11):
        module = import_module(model_name)
        getattr(module, model_name)().summary()


if __name__ == "__main__":
    parent_directory = Path(__file__).resolve().parent
    # print("Parent path to the current file:", parent_directory)
    model_name = "demonet"
    model_file_path = parent_directory / (model_name + ".py")
    # print("Model path:", model_file_path)

    run(model_name)
    print("Model demo DONE")
    time.sleep(1)

    for i in range(10):

        new_file_path = parent_directory / (model_name + "_{}.py".format(i))
        with Path.open(model_file_path, "r") as file:
            content = file.read()

        new_content = content.replace(model_name, (model_name + "_{}".format(i)))

        with Path.open(new_file_path, "w") as file:
            file.write(new_content)

        run((model_name + "_{}".format(i)))
        print("Model {} DONE".format(i))
        time.sleep(1)

```
**demonet.py** 
```c
import tensorflow as tf
from tensorflow import keras


# lenet
def demonet(label_num=10, input_shape=(28, 28, 1)):
    input_tensor = keras.Input(shape=input_shape)

    x = keras.layers.Conv2D(filters=6, kernel_size=6, strides=1, activation="relu", padding="same")(input_tensor)
    x = keras.layers.MaxPool2D(pool_size=2, strides=2)(x)

    x = keras.layers.Conv2D(filters=16, kernel_size=5, strides=1, activation="relu", padding="same")(x)
    x = keras.layers.MaxPool2D(pool_size=2, strides=2)(x)

    x = keras.layers.Conv2D(filters=32, kernel_size=5, activation="relu", padding="same")(x)
    x = keras.layers.MaxPool2D(pool_size=2, strides=2)(x)

    x = keras.layers.Flatten()(x)
    x = keras.layers.Dense(units=200, activation="relu")(x)
    output_tensor = keras.layers.Flatten()(keras.layers.Dense(units=label_num, activation="softmax")(x))

    model = keras.models.Model(inputs=input_tensor, outputs=output_tensor)

    return model

```
You can try to run main.py Hi,

I was able to run the script using Tensorflow 2.13 in my local environment.
It was able to generate the multiple files as per the script.
Below is the screenshot for reference.

Could you please let me know if that is what you are expecting, if not, could you please explain what is the expected behavior? Thanks!

<img width="1728" alt="image" src="https://github.com/tensorflow/tensorflow/assets/73069040/6a71ddfd-9689-44ae-b7d4-52f3ca4a79af">
 > Hi,
> 
> I was able to run the script using Tensorflow 2.13 in my local environment. It was able to generate the multiple files as per the script. Below is the screenshot for reference.
> 
> Could you please let me know if that is what you are expecting, if not, could you please explain what is the expected behavior? Thanks!
> 
> <img alt="image" width="1728" src="https://user-images.githubusercontent.com/73069040/254672704-6a71ddfd-9689-44ae-b7d4-52f3ca4a79af.png">

Hello, the machine configuration in my laboratory is as follows:

```
GPU
NVIDIA GeForce RTX 2080 Ti
Floating point operations per second
13.13 TFLOPS
Graphics card memory
11 GB
GPU bandwidth
616.00 GB/s
Number of channels
16
PCIE bandwidth
15.75 GB/s
Number of cpus available
6
Available memory
62 GB
Hard disk
Samsung SSD 860
Hard disk bandwidth
380.12 MB/s
Available space
100 GB
CPU model
Intel(R) Xeon(R) CPU E5-2678 v3 @ 2.50GHz
```

In running the following `demonet.py` will produce ` tensorflow. Python. Framework. Errors_impl. ResourceExhaustedError: failed to allocate memory`. 

We want to be able to clear the previous run of the model after each run so that it can run normally. If you set a breakpoint in the main function and observe the graphics card memory, you will find that the size of the graphics card memory in the process continues to grow until it reaches a peak. We have tried many ways to solve this problem, such as `tf.keras.backend.clear_session()`but it doesn't work. So we hope to obtain a better solution with this isuue.

```py
# demonet.py
import tensorflow as tf
from tensorflow import keras


def demonet(class_num=1000, input_shape=(224, 224, 3)):
    input_tensor = keras.Input(shape=input_shape)

    x = keras.layers.GlobalAveragePooling2D(keepdims=True)(input_tensor)
    x = keras.layers.BatchNormalization(moving_mean_initializer="zeros")(x)
    x = keras.layers.ZeroPadding2D(padding=(1, 4))(x)
    x = keras.layers.BatchNormalization(scale=True)(x)
    x = inceptionA_1(x, 128, 128, 128)
    x = inceptionA_2(x, 256, 256, 256)
    x = inceptionA_3(x, 728, 728, 728)
    x = inceptionB_1(x)
    x = inceptionB_2(x)
    x = inceptionB_3(x)
    x = inceptionB_4(x)
    x = inceptionB_5(x)
    x = inceptionB_6(x)
    x = inceptionB_7(x)
    x = inceptionB_8(x)
    x = inceptionA_4(x, 1024, 728, 1024)
    x = keras.layers.SeparableConv2D(filters=1536, kernel_size=[1,129], padding="same", activation="relu", use_bias=False, depth_multiplier=197)(x)

    output_tensor = keras.layers.Dense(units=class_num, activation="softmax")(keras.layers.Flatten()(x))
    model = keras.models.Model(inputs=input_tensor, outputs=output_tensor)

    return model


def inceptionA_1(inputs, filters1, filters2, filters3):
    residual = keras.layers.Conv2D(filters=filters1, kernel_size=(1,1), strides=(2,2), padding="same", use_bias=False, activation="elu")(inputs)
    residual = keras.layers.BatchNormalization(epsilon=0.9364761010664258)(residual)

    x = keras.layers.SeparableConv2D(filters=filters2, kernel_size=(3,3), padding="same", activation="relu", use_bias=False, activity_regularizer=None)(inputs)
    x = keras.layers.BatchNormalization()(x)
    x = keras.layers.SeparableConv2D(filters=filters3, kernel_size=(3, 3), padding="same")(x)
    x = keras.layers.BatchNormalization()(x)

    x = keras.layers.MaxPool2D(pool_size=(3, 3), strides=(2, 2), padding="same")(x)

    # reshape
    target_height = inputs.shape[1]
    target_width = inputs.shape[2]
    x = keras.layers.Lambda(lambda x: tf.image.resize(x, (target_height, target_width)))(x)
    residual = keras.layers.Lambda(lambda x: tf.image.resize(x, (target_height, target_width)))(residual)

    outputs = keras.layers.add([x, residual])
    return outputs


def inceptionA_2(inputs, filters1, filters2, filters3):
    residual = keras.layers.Conv2D(filters=filters1, kernel_size=(1,1), strides=4, padding="same", use_bias=False)(inputs)
    residual = keras.layers.Softmax(axis=2)(residual)

    x = keras.layers.MaxPool2D(padding="same", strides=(3, 1))(inputs)
    x = keras.layers.BatchNormalization()(x)
    x = keras.layers.SeparableConv2D(filters=filters3, kernel_size=(3, 3), padding="same")(x)
    x = keras.layers.BatchNormalization()(x)

    x = keras.layers.MaxPool2D(pool_size=(3, 3), strides=(2, 2), padding="same")(x)

    # reshape
    target_height = inputs.shape[1]
    target_width = inputs.shape[2]
    x = keras.layers.Lambda(lambda x: tf.image.resize(x, (target_height, target_width)))(x)
    residual = keras.layers.Lambda(lambda x: tf.image.resize(x, (target_height, target_width)))(residual)

    outputs = keras.layers.add([x, residual])
    return outputs


def inceptionA_3(inputs, filters1, filters2, filters3):
    residual = keras.layers.Conv2D(filters=filters1, kernel_size=(1,1), strides=[2, 5], padding="same", use_bias=False)(inputs)
    residual = keras.layers.BatchNormalization(center=False)(residual)

    x = keras.layers.SeparableConv2D(filters=filters2, kernel_size=(3,3), padding="same", activation="relu", use_bias=False, pointwise_constraint=None)(inputs)
    x = keras.layers.BatchNormalization()(x)
    x = keras.layers.SeparableConv2D(filters=filters3, kernel_size=(3, 3), padding="same")(x)
    x = keras.layers.BatchNormalization()(x)

    x = keras.layers.MaxPool2D(pool_size=(3, 3), strides=(2, 2), padding="same")(x)

    # reshape
    target_height = inputs.shape[1]
    target_width = inputs.shape[2]
    x = keras.layers.Lambda(lambda x: tf.image.resize(x, (target_height, target_width)))(x)
    residual = keras.layers.Lambda(lambda x: tf.image.resize(x, (target_height, target_width)))(residual)

    outputs = keras.layers.add([x, residual])
    return outputs


def inceptionB_1(inputs):
    residual = inputs
    x = keras.layers.SeparableConv2D(filters=728, kernel_size=128, padding="same", activation="relu", use_bias=False)(inputs)
    x = keras.layers.BatchNormalization(moving_mean_initializer="zeros")(x)
    x = keras.layers.LayerNormalization(epsilon=0.4106811257892822)(x)
    x = keras.layers.BatchNormalization()(x)
    x = keras.layers.SeparableConv2D(filters=728, kernel_size=(3, 3), padding="same", activation="relu", use_bias=False)(x)
    x = keras.layers.BatchNormalization()(x)

    # reshape
    target_height = inputs.shape[1]
    target_width = inputs.shape[2]
    x = keras.layers.Lambda(lambda x: tf.image.resize(x, (target_height, target_width)))(x)
    residual = keras.layers.Lambda(lambda x: tf.image.resize(x, (target_height, target_width)))(residual)

    outputs = keras.layers.add([x, residual])
    return outputs


def inceptionB_2(inputs):
    residual = inputs
    x = keras.layers.SeparableConv2D(filters=728, kernel_size=(3,3), padding="same", activation="relu", use_bias=False, dilation_rate=[3, 1])(inputs)
    x = keras.layers.BatchNormalization(moving_variance_initializer="ones")(x)
    x = keras.layers.SeparableConv2D(filters=728, kernel_size=(3,3), padding="same", activation="relu", use_bias=False)(x)
    x = keras.layers.BatchNormalization()(x)
    x = keras.layers.SeparableConv2D(filters=728, kernel_size=(3, 3), padding="same", activation="relu", use_bias=False)(x)
    x = keras.layers.BatchNormalization()(x)

    # reshape
    target_height = inputs.shape[1]
    target_width = inputs.shape[2]
    x = keras.layers.Lambda(lambda x: tf.image.resize(x, (target_height, target_width)))(x)
    residual = keras.layers.Lambda(lambda x: tf.image.resize(x, (target_height, target_width)))(residual)

    outputs = keras.layers.add([x, residual])
    return outputs


def inceptionB_3(inputs):
    residual = inputs
    x = keras.layers.SeparableConv2D(filters=728, kernel_size=(3,3), padding="valid", activation="relu", use_bias=False)(inputs)
    x = keras.layers.BatchNormalization(moving_mean_initializer="zeros")(x)
    x = keras.layers.Activation(activation="hard_sigmoid")(x)
    x = keras.layers.BatchNormalization()(x)
    x = keras.layers.SeparableConv2D(filters=728, kernel_size=(3, 3), padding="same", activation="relu", use_bias=False)(x)
    x = keras.layers.BatchNormalization()(x)

    # reshape
    target_height = inputs.shape[1]
    target_width = inputs.shape[2]
    x = keras.layers.Lambda(lambda x: tf.image.resize(x, (target_height, target_width)))(x)
    residual = keras.layers.Lambda(lambda x: tf.image.resize(x, (target_height, target_width)))(residual)

    outputs = keras.layers.add([x, residual])
    return outputs


def inceptionB_4(inputs):
    residual = inputs
    x = keras.layers.SeparableConv2D(filters=728, kernel_size=(3,3), padding="same", activation="relu", use_bias=False, depth_multiplier=40)(inputs)
    x = keras.layers.BatchNormalization(scale=False)(x)
    x = keras.layers.Conv2D(filters=728, kernel_size=(3,3), padding="same", activation="relu", use_bias=False, bias_initializer="zeros")(x)
    x = keras.layers.BatchNormalization()(x)
    x = keras.layers.SeparableConv2D(filters=728, kernel_size=(3, 3), padding="same", activation="relu", use_bias=False)(x)
    x = keras.layers.BatchNormalization()(x)

    # reshape
    target_height = inputs.shape[1]
    target_width = inputs.shape[2]
    x = keras.layers.Lambda(lambda x: tf.image.resize(x, (target_height, target_width)))(x)
    residual = keras.layers.Lambda(lambda x: tf.image.resize(x, (target_height, target_width)))(residual)

    outputs = keras.layers.add([x, residual])
    return outputs


def inceptionB_5(inputs):
    residual = inputs
    x = keras.layers.Conv2D(filters=728, kernel_size=(3,3), padding="valid", activation="relu", use_bias=False)(inputs)
    x = keras.layers.BatchNormalization(moving_variance_initializer="ones")(x)
    x = keras.layers.SeparableConv2D(filters=728, kernel_size=(3,3), padding="same", activation="softmax", use_bias=False)(x)
    x = keras.layers.BatchNormalization()(x)
    x = keras.layers.SeparableConv2D(filters=728, kernel_size=(3, 3), padding="same", activation="relu", use_bias=False)(x)
    x = keras.layers.BatchNormalization()(x)

    # reshape
    target_height = inputs.shape[1]
    target_width = inputs.shape[2]
    x = keras.layers.Lambda(lambda x: tf.image.resize(x, (target_height, target_width)))(x)
    residual = keras.layers.Lambda(lambda x: tf.image.resize(x, (target_height, target_width)))(residual)

    outputs = keras.layers.add([x, residual])
    return outputs


def inceptionB_6(inputs):
    residual = inputs
    x = keras.layers.Dense(activation="relu", use_bias=False, units=8102)(inputs)
    x = keras.layers.BatchNormalization(scale=False)(x)
    x = keras.layers.BatchNormalization(center=False)(x)
    x = keras.layers.BatchNormalization()(x)
    x = keras.layers.SeparableConv2D(filters=728, kernel_size=(3, 3), padding="same", activation="relu", use_bias=False)(x)
    x = keras.layers.BatchNormalization()(x)

    # reshape
    target_height = inputs.shape[1]
    target_width = inputs.shape[2]
    x = keras.layers.Lambda(lambda x: tf.image.resize(x, (target_height, target_width)))(x)
    residual = keras.layers.Lambda(lambda x: tf.image.resize(x, (target_height, target_width)))(residual)

    outputs = keras.layers.add([x, residual])
    return outputs


def inceptionB_7(inputs):
    residual = inputs
    x = keras.layers.SeparableConv2D(filters=728, kernel_size=(3,3), padding="same", activation="relu", use_bias=False)(inputs)
    x = keras.layers.BatchNormalization(scale=False)(x)
    x = keras.layers.Conv2D(filters=728, kernel_size=(3,3), padding="same", activation="relu", use_bias=False, dilation_rate=5)(x)
    x = keras.layers.BatchNormalization()(x)
    x = keras.layers.SeparableConv2D(filters=728, kernel_size=(3, 3), padding="same", activation="relu", use_bias=False)(x)
    x = keras.layers.BatchNormalization()(x)

    # reshape
    target_height = inputs.shape[1]
    target_width = inputs.shape[2]
    x = keras.layers.Lambda(lambda x: tf.image.resize(x, (target_height, target_width)))(x)
    residual = keras.layers.Lambda(lambda x: tf.image.resize(x, (target_height, target_width)))(residual)

    outputs = keras.layers.add([x, residual])
    return outputs


def inceptionB_8(inputs):
    residual = inputs
    x = keras.layers.SeparableConv2D(filters=728, kernel_size=104, padding="same", activation="relu", use_bias=False)(inputs)
    x = keras.layers.Dropout(rate=0.7893636098463078, noise_shape=None)(x)
    x = keras.layers.Conv2DTranspose(filters=728, kernel_size=(3,3), padding="same", activation="relu", use_bias=False, strides=(1, 5))(x)
    x = keras.layers.BatchNormalization()(x)
    x = keras.layers.SeparableConv2D(filters=728, kernel_size=(3, 3), padding="same", activation="relu", use_bias=False)(x)
    x = keras.layers.BatchNormalization()(x)

    # reshape
    target_height = inputs.shape[1]
    target_width = inputs.shape[2]
    x = keras.layers.Lambda(lambda x: tf.image.resize(x, (target_height, target_width)))(x)
    residual = keras.layers.Lambda(lambda x: tf.image.resize(x, (target_height, target_width)))(residual)

    outputs = keras.layers.add([x, residual])
    return outputs


def inceptionA_4(inputs, filters1, filters2, filters3):
    residual = keras.layers.Conv2DTranspose(filters=filters1, kernel_size=(1,1), strides=(2,2), padding="same", use_bias=False)(inputs)
    residual = keras.layers.BatchNormalization(momentum=0.9838259077650016)(residual)

    x = keras.layers.SeparableConv2D(filters=filters2, kernel_size=(3,3), padding="same", activation="exponential", use_bias=False)(inputs)
    x = keras.layers.BatchNormalization()(x)
    x = keras.layers.SeparableConv2D(filters=filters3, kernel_size=(3, 3), padding="same")(x)
    x = keras.layers.BatchNormalization()(x)

    x = keras.layers.MaxPool2D(pool_size=(3, 3), strides=(2, 2), padding="same")(x)

    # reshape
    target_height = inputs.shape[1]
    target_width = inputs.shape[2]
    x = keras.layers.Lambda(lambda x: tf.image.resize(x, (target_height, target_width)))(x)
    residual = keras.layers.Lambda(lambda x: tf.image.resize(x, (target_height, target_width)))(residual)

    outputs = keras.layers.add([x, residual])
    return outputs

```

```py
# main.py
import os
import time
from importlib import import_module
from pathlib import Path
import tensorflow as tf


def run(model_name):
    module = import_module(model_name)
    getattr(module, model_name)()
    # getattr(module, model_name)().summary()
    tf.keras.backend.clear_session()


if __name__ == "__main__":
    parent_directory = Path(__file__).resolve().parent

    model_name = "demonet"
    model_file_path = parent_directory / (model_name + ".py")

    run(model_name)
    print("Model demo DONE")

    for i in range(1000):

        new_file_path = parent_directory / (model_name + "_{}.py".format(i))
        with Path.open(model_file_path, "r") as file:
            content = file.read()

        new_content = content.replace(model_name, (model_name + "_{}".format(i)))

        with Path.open(new_file_path, "w") as file:
            file.write(new_content)

        run((model_name + "_{}".format(i)))
        print("Model {} DONE".format(i))
        time.sleep(1)

```

And from the output of the named suffixes in `summary()`, the process of convolution, pooling and so on is also cumulative, not a set of named suffixes for a model.

Here I posted my error details

```
2023-07-20 05:51:24.067591: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a6800 of size 256 next 1134
2023-07-20 05:51:24.067598: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a6900 of size 256 next 1135
2023-07-20 05:51:24.067606: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a6a00 of size 256 next 1136
2023-07-20 05:51:24.067612: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a6b00 of size 256 next 1137
2023-07-20 05:51:24.067619: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a6c00 of size 256 next 1138
2023-07-20 05:51:24.067626: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a6d00 of size 512 next 1139
2023-07-20 05:51:24.067633: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a6f00 of size 512 next 1140
2023-07-20 05:51:24.067640: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a7100 of size 512 next 1143
2023-07-20 05:51:24.067647: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a7300 of size 512 next 1144
2023-07-20 05:51:24.067654: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a7500 of size 256 next 1145
2023-07-20 05:51:24.067661: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a7600 of size 256 next 1146
2023-07-20 05:51:24.067668: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a7700 of size 768 next 1141
2023-07-20 05:51:24.067674: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a7a00 of size 1536 next 1142
2023-07-20 05:51:24.067679: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a8000 of size 512 next 1147
2023-07-20 05:51:24.067686: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a8200 of size 512 next 1150
2023-07-20 05:51:24.067693: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a8400 of size 512 next 1151
2023-07-20 05:51:24.067701: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a8600 of size 512 next 1152
2023-07-20 05:51:24.067708: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a8800 of size 512 next 1156
2023-07-20 05:51:24.067715: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a8a00 of size 512 next 1148
2023-07-20 05:51:24.067722: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a8c00 of size 1536 next 1149
2023-07-20 05:51:24.067729: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a9200 of size 512 next 1157
2023-07-20 05:51:24.067736: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a9400 of size 512 next 1158
2023-07-20 05:51:24.067742: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a9600 of size 512 next 1160
2023-07-20 05:51:24.067750: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a9800 of size 512 next 1161
2023-07-20 05:51:24.067755: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a9a00 of size 512 next 1162
2023-07-20 05:51:24.067762: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a9c00 of size 512 next 1163
2023-07-20 05:51:24.067769: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3a9e00 of size 1024 next 1170
2023-07-20 05:51:24.067776: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3aa200 of size 3072 next 1172
2023-07-20 05:51:24.067783: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3aae00 of size 3840 next 279
2023-07-20 05:51:24.067790: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3abd00 of size 4096 next 301
2023-07-20 05:51:24.067797: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3acd00 of size 4096 next 295
2023-07-20 05:51:24.067803: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3add00 of size 512 next 310
2023-07-20 05:51:24.067810: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3adf00 of size 512 next 296
2023-07-20 05:51:24.067817: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3ae100 of size 512 next 350
2023-07-20 05:51:24.067824: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3ae300 of size 1536 next 189
2023-07-20 05:51:24.067830: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3ae900 of size 512 next 329
2023-07-20 05:51:24.067837: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3aeb00 of size 512 next 319
2023-07-20 05:51:24.067844: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3aed00 of size 512 next 248
2023-07-20 05:51:24.067851: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3aef00 of size 1536 next 332
2023-07-20 05:51:24.067857: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3af500 of size 512 next 338
2023-07-20 05:51:24.067864: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3af700 of size 512 next 270
2023-07-20 05:51:24.067871: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3af900 of size 512 next 339
2023-07-20 05:51:24.067878: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3afb00 of size 512 next 321
2023-07-20 05:51:24.067884: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3afd00 of size 512 next 322
2023-07-20 05:51:24.067891: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3aff00 of size 512 next 297
2023-07-20 05:51:24.067898: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3b0100 of size 512 next 351
2023-07-20 05:51:24.067905: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3b0300 of size 1024 next 432
2023-07-20 05:51:24.067912: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3b0700 of size 3072 next 245
2023-07-20 05:51:24.067919: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3b1300 of size 4352 next 278
2023-07-20 05:51:24.067926: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3b2400 of size 4608 next 1153
2023-07-20 05:51:24.067933: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3b3600 of size 1024 next 1167
2023-07-20 05:51:24.067939: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3b3a00 of size 1024 next 1168
2023-07-20 05:51:24.067946: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3b3e00 of size 1024 next 1169
2023-07-20 05:51:24.067953: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3b4200 of size 1536 next 1164
2023-07-20 05:51:24.067960: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3b4800 of size 4608 next 1165
2023-07-20 05:51:24.067966: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3b5a00 of size 3072 next 1173
2023-07-20 05:51:24.067973: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3b6600 of size 3072 next 1174
2023-07-20 05:51:24.067980: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3b7200 of size 3072 next 1175
2023-07-20 05:51:24.067987: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3b7e00 of size 5120 next 306
2023-07-20 05:51:24.067994: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3b9200 of size 65536 next 328
2023-07-20 05:51:24.068001: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d3c9200 of size 226304 next 313
2023-07-20 05:51:24.068007: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d400600 of size 131072 next 1159
2023-07-20 05:51:24.068014: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d420600 of size 131072 next 1166
2023-07-20 05:51:24.068021: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d440600 of size 3072 next 1231
2023-07-20 05:51:24.068028: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d441200 of size 3072 next 1232
2023-07-20 05:51:24.068035: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d441e00 of size 3072 next 1233
2023-07-20 05:51:24.068042: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d442a00 of size 3072 next 1234
2023-07-20 05:51:24.068048: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d443600 of size 3072 next 1238
2023-07-20 05:51:24.068055: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d444200 of size 3072 next 1239
2023-07-20 05:51:24.068062: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d444e00 of size 3072 next 1240
2023-07-20 05:51:24.068069: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d445a00 of size 4864 next 1228
2023-07-20 05:51:24.068076: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d446d00 of size 26368 next 1229
2023-07-20 05:51:24.068082: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d44d400 of size 3072 next 1256
2023-07-20 05:51:24.068089: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d44e000 of size 3072 next 1257
2023-07-20 05:51:24.068096: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d44ec00 of size 46592 next 1247
2023-07-20 05:51:24.068103: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d45a200 of size 26368 next 1248
2023-07-20 05:51:24.068110: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d460900 of size 3072 next 1261
2023-07-20 05:51:24.068117: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d461500 of size 3072 next 1262
2023-07-20 05:51:24.068124: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d462100 of size 3072 next 1263
2023-07-20 05:51:24.068130: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d462d00 of size 3072 next 1264
2023-07-20 05:51:24.068137: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d463900 of size 3072 next 1267
2023-07-20 05:51:24.068144: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d464500 of size 3072 next 1268
2023-07-20 05:51:24.068151: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d465100 of size 3072 next 1269
2023-07-20 05:51:24.068158: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d465d00 of size 4864 next 1258
2023-07-20 05:51:24.068164: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d467000 of size 26368 next 1259
2023-07-20 05:51:24.068172: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d46d700 of size 41216 next 154
2023-07-20 05:51:24.068179: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f7d477800 of size 47710208 next 453
2023-07-20 05:51:24.068186: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f801f7800 of size 31496192 next 697
2023-07-20 05:51:24.068193: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f82001000 of size 31496192 next 881
2023-07-20 05:51:24.068200: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f83e0a800 of size 55185408 next 120
2023-07-20 05:51:24.068207: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f872ab800 of size 170836992 next 367
2023-07-20 05:51:24.068214: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91597c00 of size 2119936 next 1216
2023-07-20 05:51:24.068222: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f9179d500 of size 2119936 next 1220
2023-07-20 05:51:24.068229: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f919a2e00 of size 2119936 next 1230
2023-07-20 05:51:24.068236: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91ba8700 of size 745472 next 275
2023-07-20 05:51:24.068243: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91c5e700 of size 131072 next 6
2023-07-20 05:51:24.068250: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91c7e700 of size 131072 next 354
2023-07-20 05:51:24.068258: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91c9e700 of size 26368 next 669
2023-07-20 05:51:24.068265: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91ca4e00 of size 26368 next 442
2023-07-20 05:51:24.068272: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cab500 of size 3072 next 464
2023-07-20 05:51:24.068278: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cac100 of size 3072 next 421
2023-07-20 05:51:24.068285: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cacd00 of size 3072 next 420
2023-07-20 05:51:24.068292: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cad900 of size 3072 next 422
2023-07-20 05:51:24.068299: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cae500 of size 3072 next 410
2023-07-20 05:51:24.068306: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91caf100 of size 3072 next 114
2023-07-20 05:51:24.068313: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cafd00 of size 3072 next 417
2023-07-20 05:51:24.068319: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cb0900 of size 4864 next 403
2023-07-20 05:51:24.068327: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cb1c00 of size 26368 next 528
2023-07-20 05:51:24.068333: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cb8300 of size 3072 next 425
2023-07-20 05:51:24.068340: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cb8f00 of size 3072 next 413
2023-07-20 05:51:24.068347: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cb9b00 of size 3072 next 414
2023-07-20 05:51:24.068353: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cba700 of size 3072 next 558
2023-07-20 05:51:24.068360: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cbb300 of size 3072 next 426
2023-07-20 05:51:24.068367: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cbbf00 of size 3072 next 427
2023-07-20 05:51:24.068374: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cbcb00 of size 3072 next 454
2023-07-20 05:51:24.068381: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cbd700 of size 4864 next 416
2023-07-20 05:51:24.068388: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cbea00 of size 26368 next 412
2023-07-20 05:51:24.068395: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cc5100 of size 3072 next 387
2023-07-20 05:51:24.068402: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cc5d00 of size 3072 next 491
2023-07-20 05:51:24.068408: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cc6900 of size 3072 next 492
2023-07-20 05:51:24.068415: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cc7500 of size 3072 next 446
2023-07-20 05:51:24.068422: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cc8100 of size 3072 next 448
2023-07-20 05:51:24.068429: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cc8d00 of size 3072 next 108
2023-07-20 05:51:24.068436: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cc9900 of size 3072 next 389
2023-07-20 05:51:24.068443: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cca500 of size 4864 next 394
2023-07-20 05:51:24.068450: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91ccb800 of size 26368 next 393
2023-07-20 05:51:24.068457: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cd1f00 of size 3072 next 5
2023-07-20 05:51:24.068464: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cd2b00 of size 3072 next 391
2023-07-20 05:51:24.068471: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cd3700 of size 3072 next 445
2023-07-20 05:51:24.068478: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cd4300 of size 3072 next 508
2023-07-20 05:51:24.068483: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cd4f00 of size 3072 next 524
2023-07-20 05:51:24.068490: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cd5b00 of size 3072 next 472
2023-07-20 05:51:24.068497: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cd6700 of size 3072 next 559
2023-07-20 05:51:24.068504: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cd7300 of size 4864 next 428
2023-07-20 05:51:24.068511: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cd8600 of size 26368 next 447
2023-07-20 05:51:24.068518: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cded00 of size 57856 next 223
2023-07-20 05:51:24.068526: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91cecf00 of size 291840 next 124
2023-07-20 05:51:24.068662: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d34300 of size 3072 next 78
2023-07-20 05:51:24.068669: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d34f00 of size 3072 next 370
2023-07-20 05:51:24.068676: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d35b00 of size 3072 next 379
2023-07-20 05:51:24.068683: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d36700 of size 3072 next 378
2023-07-20 05:51:24.068690: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d37300 of size 4096 next 47
2023-07-20 05:51:24.068697: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d38300 of size 4096 next 365
2023-07-20 05:51:24.068704: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d39300 of size 5888 next 148
2023-07-20 05:51:24.068712: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d3aa00 of size 26368 next 80
2023-07-20 05:51:24.068718: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d41100 of size 4096 next 366
2023-07-20 05:51:24.068725: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d42100 of size 4096 next 128
2023-07-20 05:51:24.068732: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d43100 of size 4096 next 71
2023-07-20 05:51:24.068739: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d44100 of size 256 next 69
2023-07-20 05:51:24.068746: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d44200 of size 256 next 32
2023-07-20 05:51:24.068753: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d44300 of size 256 next 49
2023-07-20 05:51:24.068760: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d44400 of size 256 next 90
2023-07-20 05:51:24.068767: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d44500 of size 256 next 36
2023-07-20 05:51:24.068774: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d44600 of size 256 next 66
2023-07-20 05:51:24.068781: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d44700 of size 256 next 121
2023-07-20 05:51:24.068788: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d44800 of size 512 next 38
2023-07-20 05:51:24.068795: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d44a00 of size 512 next 109
2023-07-20 05:51:24.068801: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d44c00 of size 512 next 546
2023-07-20 05:51:24.068808: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d44e00 of size 512 next 452
2023-07-20 05:51:24.068815: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d45000 of size 256 next 175
2023-07-20 05:51:24.068822: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d45100 of size 256 next 369
2023-07-20 05:51:24.068829: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d45200 of size 768 next 106
2023-07-20 05:51:24.068836: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d45500 of size 1536 next 60
2023-07-20 05:51:24.068843: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d45b00 of size 512 next 17
2023-07-20 05:51:24.068850: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d45d00 of size 512 next 238
2023-07-20 05:51:24.068857: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d45f00 of size 512 next 62
2023-07-20 05:51:24.068864: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d46100 of size 512 next 7
2023-07-20 05:51:24.068870: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d46300 of size 512 next 360
2023-07-20 05:51:24.068877: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d46500 of size 512 next 115
2023-07-20 05:51:24.068884: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d46700 of size 1536 next 42
2023-07-20 05:51:24.068891: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d46d00 of size 512 next 488
2023-07-20 05:51:24.068898: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d46f00 of size 512 next 100
2023-07-20 05:51:24.068905: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d47100 of size 512 next 107
2023-07-20 05:51:24.068912: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d47300 of size 512 next 67
2023-07-20 05:51:24.068919: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d47500 of size 768 next 77
2023-07-20 05:51:24.068926: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d47800 of size 26368 next 24
2023-07-20 05:51:24.068933: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d4df00 of size 3072 next 358
2023-07-20 05:51:24.068940: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d4eb00 of size 3072 next 126
2023-07-20 05:51:24.068990: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d4f700 of size 3072 next 215
2023-07-20 05:51:24.068997: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d50300 of size 3072 next 253
2023-07-20 05:51:24.069004: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d50f00 of size 3072 next 388
2023-07-20 05:51:24.069011: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d51b00 of size 3072 next 435
2023-07-20 05:51:24.069018: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d52700 of size 9216 next 380
2023-07-20 05:51:24.069025: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d54b00 of size 3072 next 808
2023-07-20 05:51:24.069032: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d55700 of size 3072 next 659
2023-07-20 05:51:24.069039: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d56300 of size 3072 next 8
2023-07-20 05:51:24.069046: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d56f00 of size 3072 next 493
2023-07-20 05:51:24.069053: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d57b00 of size 3072 next 384
2023-07-20 05:51:24.069060: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d58700 of size 3072 next 374
2023-07-20 05:51:24.069067: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d59300 of size 3072 next 331
2023-07-20 05:51:24.069073: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d59f00 of size 3072 next 409
2023-07-20 05:51:24.069080: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d5ab00 of size 3072 next 439
2023-07-20 05:51:24.069087: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d5b700 of size 3072 next 440
2023-07-20 05:51:24.069094: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d5c300 of size 3072 next 133
2023-07-20 05:51:24.069101: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d5cf00 of size 3072 next 441
2023-07-20 05:51:24.069108: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d5db00 of size 3072 next 34
2023-07-20 05:51:24.069115: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d5e700 of size 3072 next 207
2023-07-20 05:51:24.069122: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d5f300 of size 3072 next 26
2023-07-20 05:51:24.069128: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d5ff00 of size 3072 next 177
2023-07-20 05:51:24.069135: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d60b00 of size 3584 next 315
2023-07-20 05:51:24.069142: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d61900 of size 50688 next 542
2023-07-20 05:51:24.069149: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d6df00 of size 65536 next 136
2023-07-20 05:51:24.069156: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d7df00 of size 3072 next 390
2023-07-20 05:51:24.069163: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d7eb00 of size 3072 next 429
2023-07-20 05:51:24.069170: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d7f700 of size 46592 next 740
2023-07-20 05:51:24.069177: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d8ad00 of size 26368 next 385
2023-07-20 05:51:24.069184: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d91400 of size 26368 next 724
2023-07-20 05:51:24.069191: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d97b00 of size 3072 next 93
2023-07-20 05:51:24.069198: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d98700 of size 3072 next 86
2023-07-20 05:51:24.069706: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d99300 of size 3072 next 94
2023-07-20 05:51:24.069715: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d99f00 of size 3072 next 88
2023-07-20 05:51:24.069722: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d9ab00 of size 3072 next 362
2023-07-20 05:51:24.069729: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d9b700 of size 3072 next 363
2023-07-20 05:51:24.069736: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d9c300 of size 3072 next 549
2023-07-20 05:51:24.069742: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d9cf00 of size 4864 next 431
2023-07-20 05:51:24.069750: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91d9e200 of size 26368 next 399
2023-07-20 05:51:24.069757: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91da4900 of size 3072 next 371
2023-07-20 05:51:24.069763: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91da5500 of size 3072 next 28
2023-07-20 05:51:24.069770: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91da6100 of size 3072 next 372
2023-07-20 05:51:24.069777: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91da6d00 of size 3072 next 373
2023-07-20 05:51:24.069784: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91da7900 of size 3072 next 419
2023-07-20 05:51:24.069791: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91da8500 of size 3072 next 418
2023-07-20 05:51:24.069798: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91da9100 of size 3072 next 392
2023-07-20 05:51:24.069805: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91da9d00 of size 3072 next 406
2023-07-20 05:51:24.069811: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91daa900 of size 3072 next 459
2023-07-20 05:51:24.069818: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91dab500 of size 3072 next 423
2023-07-20 05:51:24.069825: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91dac100 of size 3072 next 415
2023-07-20 05:51:24.069833: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91dacd00 of size 4864 next 1246
2023-07-20 05:51:24.069840: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91dae000 of size 2119936 next 1249
2023-07-20 05:51:24.069847: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f91fb3900 of size 2119936 next 1260
2023-07-20 05:51:24.069854: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f921b9200 of size 2119936 next 1266
2023-07-20 05:51:24.069861: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f923beb00 of size 2119936 next 1287
2023-07-20 05:51:24.069868: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f925c4400 of size 2119936 next 1300
2023-07-20 05:51:24.069875: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f927c9d00 of size 2119936 next 1305
2023-07-20 05:51:24.069882: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f929cf600 of size 2119936 next 646
2023-07-20 05:51:24.069889: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f92bd4f00 of size 2119936 next 693
2023-07-20 05:51:24.069896: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f92dda800 of size 2119936 next 683
2023-07-20 05:51:24.069903: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f92fe0100 of size 2119936 next 1018
2023-07-20 05:51:24.069910: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f931e5a00 of size 2119936 next 704
2023-07-20 05:51:24.069916: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f933eb300 of size 2981888 next 485
2023-07-20 05:51:24.069924: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f936c3300 of size 5497856 next 584
2023-07-20 05:51:24.069931: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f93c01700 of size 19079424 next 569
2023-07-20 05:51:24.069938: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f94e33800 of size 19079424 next 545
2023-07-20 05:51:24.069945: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f96065900 of size 2981888 next 723
2023-07-20 05:51:24.069952: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f9633d900 of size 2981888 next 713
2023-07-20 05:51:24.069959: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f96615900 of size 2119936 next 774
2023-07-20 05:51:24.069965: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f9681b200 of size 2119936 next 761
2023-07-20 05:51:24.069972: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f96a20b00 of size 2119936 next 813
2023-07-20 05:51:24.069979: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f96c26400 of size 2119936 next 827
2023-07-20 05:51:24.069986: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f96e2bd00 of size 2119936 next 801
2023-07-20 05:51:24.069993: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f97031600 of size 1048320 next 846
2023-07-20 05:51:24.070000: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 7f9f97131500 of size 1467648 next 671
2023-07-20 05:51:24.070007: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f97297a00 of size 27773952 next 696
2023-07-20 05:51:24.070014: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f98d14600 of size 104091648 next 739
2023-07-20 05:51:24.070021: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9f9f059600 of size 84797440 next 627
2023-07-20 05:51:24.070029: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9fa4137e00 of size 23593216 next 728
2023-07-20 05:51:24.070035: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9fa57b7f00 of size 23593216 next 644
2023-07-20 05:51:24.070042: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9fa6e38000 of size 19079424 next 648
2023-07-20 05:51:24.070049: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 7f9fa806a100 of size 2119936 next 772
2023-07-20 05:51:24.070056: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9fa826fa00 of size 2119936 next 815
2023-07-20 05:51:24.070063: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9fa8475300 of size 2119936 next 833
2023-07-20 05:51:24.070070: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9fa867ac00 of size 2119936 next 824
2023-07-20 05:51:24.070077: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9fa8880500 of size 2119936 next 591
2023-07-20 05:51:24.070084: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9fa8a85e00 of size 2119936 next 899
2023-07-20 05:51:24.070092: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9fa8c8b700 of size 2119936 next 905
2023-07-20 05:51:24.070097: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9fa8e91000 of size 2119936 next 880
2023-07-20 05:51:24.070104: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 7f9fa9096900 of size 2119936 next 816
2023-07-20 05:51:24.070111: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9fa929c200 of size 19785728 next 162
2023-07-20 05:51:24.070118: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9faa57aa00 of size 256 next 163
2023-07-20 05:51:24.070125: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7f9faa57ab00 of size 1239416832 next 997
2023-07-20 05:51:24.070132: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 7f9ff437ab00 of size 563326208 next 228
2023-07-20 05:51:24.070140: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7fa015cb5800 of size 1374464 next 436
2023-07-20 05:51:24.070145: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7fa015e05100 of size 1239416832 next 314
2023-07-20 05:51:24.070153: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7fa05fc05100 of size 1239416832 next 203
2023-07-20 05:51:24.070160: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7fa0a9a05100 of size 1239416832 next 443
2023-07-20 05:51:24.070167: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7fa0f3805100 of size 1239416832 next 685
2023-07-20 05:51:24.070174: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 7fa13d605100 of size 2981888 next 908
2023-07-20 05:51:24.070181: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7fa13d8dd100 of size 2981888 next 928
2023-07-20 05:51:24.070188: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 7fa13dbb5100 of size 2981888 next 1093
2023-07-20 05:51:24.070195: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7fa13de8d100 of size 2981888 next 1015
2023-07-20 05:51:24.070201: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 7fa13e165100 of size 7151872 next 849
2023-07-20 05:51:24.070209: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7fa13e837200 of size 28630784 next 859
2023-07-20 05:51:24.070216: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7fa140385100 of size 47710208 next 878
2023-07-20 05:51:24.070223: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 7fa143105100 of size 23593216 next 812
2023-07-20 05:51:24.070230: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7fa144785200 of size 46874368 next 735
2023-07-20 05:51:24.070237: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7fa147439100 of size 165888000 next 745
2023-07-20 05:51:24.070244: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 7fa15126d100 of size 208183296 next 1003
2023-07-20 05:51:24.070251: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 7fa15d8f7100 of size 104091648 next 1006
2023-07-20 05:51:24.070259: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at 7fa163c3c100 of size 977551104 next 18446744073709551615
2023-07-20 05:51:24.070266: I tensorflow/core/common_runtime/bfc_allocator.cc:1071]      Summary of in-use Chunks by size: 
2023-07-20 05:51:24.070275: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 90 Chunks of size 256 totalling 22.5KiB
2023-07-20 05:51:24.070283: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 78 Chunks of size 512 totalling 39.0KiB
2023-07-20 05:51:24.070291: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 7 Chunks of size 768 totalling 5.2KiB
2023-07-20 05:51:24.070299: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 20 Chunks of size 1024 totalling 20.0KiB
2023-07-20 05:51:24.070306: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 1280 totalling 1.2KiB
2023-07-20 05:51:24.070314: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 15 Chunks of size 1536 totalling 22.5KiB
2023-07-20 05:51:24.070321: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 460 Chunks of size 3072 totalling 1.35MiB
2023-07-20 05:51:24.070329: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 3 Chunks of size 3584 totalling 10.5KiB
2023-07-20 05:51:24.070337: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 3840 totalling 7.5KiB
2023-07-20 05:51:24.070345: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 44 Chunks of size 4096 totalling 176.0KiB
2023-07-20 05:51:24.070351: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 3 Chunks of size 4352 totalling 12.8KiB
2023-07-20 05:51:24.070360: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 13 Chunks of size 4608 totalling 58.5KiB
2023-07-20 05:51:24.070367: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 33 Chunks of size 4864 totalling 156.8KiB
2023-07-20 05:51:24.070375: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 5120 totalling 10.0KiB
2023-07-20 05:51:24.070383: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 5376 totalling 10.5KiB
2023-07-20 05:51:24.070390: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 11 Chunks of size 5888 totalling 63.2KiB
2023-07-20 05:51:24.070398: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 6144 totalling 6.0KiB
2023-07-20 05:51:24.070405: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 4 Chunks of size 9216 totalling 36.0KiB
2023-07-20 05:51:24.070413: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 12800 totalling 12.5KiB
2023-07-20 05:51:24.070421: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 62 Chunks of size 26368 totalling 1.56MiB
2023-07-20 05:51:24.070428: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 4 Chunks of size 28160 totalling 110.0KiB
2023-07-20 05:51:24.070436: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 52 Chunks of size 32512 totalling 1.61MiB
2023-07-20 05:51:24.070444: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 39168 totalling 38.2KiB
2023-07-20 05:51:24.070451: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 41216 totalling 40.2KiB
2023-07-20 05:51:24.070459: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 42496 totalling 41.5KiB
2023-07-20 05:51:24.070467: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 43520 totalling 42.5KiB
2023-07-20 05:51:24.070475: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 45568 totalling 44.5KiB
2023-07-20 05:51:24.070482: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 5 Chunks of size 46592 totalling 227.5KiB
2023-07-20 05:51:24.070490: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 47872 totalling 46.8KiB
2023-07-20 05:51:24.070498: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 50688 totalling 99.0KiB
2023-07-20 05:51:24.070506: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 51968 totalling 50.8KiB
2023-07-20 05:51:24.070514: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 52480 totalling 51.2KiB
2023-07-20 05:51:24.070521: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 57856 totalling 56.5KiB
2023-07-20 05:51:24.070529: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 64768 totalling 63.2KiB
2023-07-20 05:51:24.070537: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 5 Chunks of size 65536 totalling 320.0KiB
2023-07-20 05:51:24.070544: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 9 Chunks of size 131072 totalling 1.12MiB
2023-07-20 05:51:24.070552: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 226304 totalling 221.0KiB
2023-07-20 05:51:24.070561: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 4 Chunks of size 291840 totalling 1.11MiB
2023-07-20 05:51:24.070569: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 410368 totalling 400.8KiB
2023-07-20 05:51:24.070577: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 6 Chunks of size 745472 totalling 4.27MiB
2023-07-20 05:51:24.070584: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 4 Chunks of size 1048320 totalling 4.00MiB
2023-07-20 05:51:24.070592: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 1071616 totalling 1.02MiB
2023-07-20 05:51:24.070600: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 1188096 totalling 2.27MiB
2023-07-20 05:51:24.070607: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 1374464 totalling 2.62MiB
2023-07-20 05:51:24.070615: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 68 Chunks of size 2119936 totalling 137.48MiB
2023-07-20 05:51:24.070623: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 2912000 totalling 5.55MiB
2023-07-20 05:51:24.070631: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 13 Chunks of size 2981888 totalling 36.97MiB
2023-07-20 05:51:24.070638: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 3656960 totalling 3.49MiB
2023-07-20 05:51:24.070646: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 3748352 totalling 3.57MiB
2023-07-20 05:51:24.070654: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 3843840 totalling 7.33MiB
2023-07-20 05:51:24.070662: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 4286464 totalling 4.09MiB
2023-07-20 05:51:24.070669: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 5031936 totalling 4.80MiB
2023-07-20 05:51:24.070677: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 5497856 totalling 5.24MiB
2023-07-20 05:51:24.070685: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 8 Chunks of size 19079424 totalling 145.56MiB
2023-07-20 05:51:24.070693: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 19785728 totalling 18.87MiB
2023-07-20 05:51:24.070701: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 8 Chunks of size 23593216 totalling 180.00MiB
2023-07-20 05:51:24.070709: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 26953472 totalling 25.70MiB
2023-07-20 05:51:24.070717: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 27773952 totalling 26.49MiB
2023-07-20 05:51:24.070724: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 27820800 totalling 26.53MiB
2023-07-20 05:51:24.070732: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 28630784 totalling 54.61MiB
2023-07-20 05:51:24.070739: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 29154560 totalling 27.80MiB
2023-07-20 05:51:24.070747: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 6 Chunks of size 31496192 totalling 180.22MiB
2023-07-20 05:51:24.070756: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 33062144 totalling 31.53MiB
2023-07-20 05:51:24.070764: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 34817792 totalling 33.20MiB
2023-07-20 05:51:24.070772: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 37611008 totalling 35.87MiB
2023-07-20 05:51:24.070779: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 46874368 totalling 44.70MiB
2023-07-20 05:51:24.070787: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 4 Chunks of size 47710208 totalling 182.00MiB
2023-07-20 05:51:24.070795: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 53301248 totalling 50.83MiB
2023-07-20 05:51:24.070802: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 55185408 totalling 52.63MiB
2023-07-20 05:51:24.070810: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 56381440 totalling 53.77MiB
2023-07-20 05:51:24.070818: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 84797440 totalling 161.74MiB
2023-07-20 05:51:24.070826: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 100119808 totalling 95.48MiB
2023-07-20 05:51:24.070833: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 3 Chunks of size 104091648 totalling 297.81MiB
2023-07-20 05:51:24.070841: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 123385856 totalling 117.67MiB
2023-07-20 05:51:24.070849: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 129020928 totalling 123.04MiB
2023-07-20 05:51:24.070857: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 157307136 totalling 150.02MiB
2023-07-20 05:51:24.070865: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 3 Chunks of size 165888000 totalling 474.61MiB
2023-07-20 05:51:24.070872: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 170836992 totalling 162.92MiB
2023-07-20 05:51:24.070880: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 190129408 totalling 181.32MiB
2023-07-20 05:51:24.070888: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 6 Chunks of size 1239416832 totalling 6.93GiB
2023-07-20 05:51:24.070896: I tensorflow/core/common_runtime/bfc_allocator.cc:1078] Sum Total of in-use chunks: 10.02GiB
2023-07-20 05:51:24.070903: I tensorflow/core/common_runtime/bfc_allocator.cc:1080] total_region_allocated_bytes_: 13791395840 memory_limit_: 13791395840 available bytes: 0 curr_region_allocation_bytes_: 27582791680
2023-07-20 05:51:24.070914: I tensorflow/core/common_runtime/bfc_allocator.cc:1086] Stats: 
Limit:                     13791395840
InUse:                     10757204480
MaxInUse:                  12095458048
NumAllocs:                        5845
MaxAllocSize:               1322475776
Reserved:                            0
PeakReserved:                        0
LargestFreeBlock:                    0

2023-07-20 05:51:24.070964: W tensorflow/core/common_runtime/bfc_allocator.cc:474] **********************_______********************___***************************************_*_______
2023-07-20 05:51:24.070991: W tensorflow/core/framework/op_kernel.cc:1733] RESOURCE_EXHAUSTED: failed to allocate memory
```

Also try on: 

```
GPU
NVIDIA A16
Floating point operations per second
2.19 TFLOPS
Graphics card memory
15 GB
GPU bandwidth
200.03 GB/s
Number of channels
16
PCIE bandwidth
31.50 GB/s
Number of cpus available
3
Available memory
14 GB
Hard disk
LOGICAL VOLUME
Hard disk bandwidth
431.04 MB/s
Available space
100 GB
CPU model
Intel(R) Xeon(R) Gold 5318Y CPU @ 2.10GHz
```

 Thanks for the detailed information, Did you try this in any other OS, or is this specific to MacOS?  > Thanks for the detailed information, Did you try this in any other OS, or is this specific to MacOS?

We also try this in Ubuntu，you can see the detailed log. Thank you!
### Issue type

Performance

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

binary

### TensorFlow version

tf2.12.0

### Custom code

Yes

### OS platform and distribution

MacOs

### Mobile device

_No response_

### Python version

3.9

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

We have many similar model building and training code files, and train multiple models consecutively through one process. But in this process, we used the memory blank and the restriction function, and neither worked.


### Standalone code to reproduce the issue

```shell
![image](https://github.com/tensorflow/tensorflow/assets/34181680/fbe4f5b6-255b-4583-b108-f476bd639574)
```

### Relevant log output

_No response_