Cuda memory error
@Utkarsha666,
Could you please try limiting GPU memory growth using any of the methods listed in this guide.
https://www.tensorflow.org/guide/gpu#limiting_gpu_memory_growth

Could you please try using the latest Tensorflow version for windows which can be installed with WSL2.
Also, from TensorFlow 2.12 CUDA and CuDNN dependency has been upgraded to 11.8 and 8.6 respectively.
Refer the document here for installation instructions https://www.tensorflow.org/install/pip and let us know if you observe the same behavior in the latest Tensorflow version. Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60401">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60401">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Performance

### Have you reproduced the bug with TF nightly?

No

### Source

binary

### Tensorflow Version

2.10

### Custom Code

Yes

### OS Platform and Distribution

Windows 11

### Mobile device

_No response_

### Python version

3.8.16

### Bazel version

_No response_

### GCC/Compiler version

visual studio

### CUDA/cuDNN version

11.2/v8.2.1.32

### GPU model and memory

GTX 1650/ System Memory:8GB

### Current Behaviour?

I am trying to train a model for binary classification, with class1 dataset= 2716 and class 2 dataset= 2164, 
The training did happened successfully but during prediction I got error. 

The code is quite short so I will paste the whole code here, 

```
import matplotlib.pyplot as plt
from tensorflow.keras.models import Model
from tensorflow.keras.layers import GlobalAveragePooling2D, Dense
from tensorflow.keras.applications import vgg16
from tensorflow.keras.optimizers import Adam, SGD
import tensorflow as tf
import scipy

import os
import cv2
from PIL import Image
import numpy as np

# i have habit of writing this on every program, since my GPU memory is only 4GB
if tf.config.list_physical_devices('GPU'):
    physical_devices = tf.config.list_physical_devices('GPU')
    tf.config.experimental.set_memory_growth(physical_devices[0], enable=True)
    tf.config.experimental.set_virtual_device_configuration(physical_devices[0], [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=12000)])


SIZE = 224

dataset = []
label = []

parasitized_images = os.listdir('Parasitized/')

for i, image_name in enumerate(parasitized_images):
    
    if (image_name.split('.')[1] == 'png'):
        image = cv2.imread('Parasitized/' + image_name)
        image = Image.fromarray(image, 'RGB')
        image = image.resize((SIZE, SIZE))
        dataset.append(np.array(image))
        label.append(1)
    
uninfected_images = os.listdir('Uninfected/')
for i, image_name in enumerate(uninfected_images):
    if (image_name.split('.')[1] == 'png'):
        image = cv2.imread('Uninfected/' + image_name)
        image = Image.fromarray(image, 'RGB')
        image = image.resize((SIZE, SIZE))
        dataset.append(np.array(image))
        label.append(0)
        
dataset = np.array(dataset)
label = np.array(label)

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(dataset, label, test_size=0.2, random_state=0)

X_train = X_train.astype('float32') / 255
X_test = X_test.astype('float32') / 255     

from tensorflow.keras.utils import to_categorical

y_train = to_categorical(y_train)
y_test = to_categorical(y_test)

def get_model(input_shape = (224, 224, 3)):
    
    with tf.device('/gpu:0'):
    
        vgg = vgg16.VGG16(weights='imagenet', include_top=False, input_shape = input_shape)
        
        for layer in vgg.layers[:-5]:
            print(layer.name)
            layer.trainable = False
            
        x = vgg.output
        x = GlobalAveragePooling2D()(x)
        x = Dense(2, activation="softmax")(x)
            
        model = Model(vgg.input, x)
            
        return model



model = get_model(input_shape = (224, 224, 3))

with tf.device('/cpu:0'):
    model.compile(loss="categorical_crossentropy",
                  optimizer = SGD(lr=0.0001, momentum=0.9), metrics=['accuracy'])

print(model.summary())
history = model.fit(X_train, y_train, batch_size=8, epochs=5, validation_data=(X_test, y_test))

#plot the training and validation accuracy and loss at each epoch
loss = history.history['loss']
val_loss = history.history['val_loss']
epochs = range(1, len(loss) + 1)
plt.plot(epochs, loss, 'y', label='Training loss')
plt.plot(epochs, val_loss, 'r', label='Validation loss')
plt.title('Training and validation loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()
plt.show()


acc = history.history['accuracy']
val_acc = history.history['val_accuracy']
plt.plot(epochs, acc, 'y', label='Training acc')
plt.plot(epochs, val_acc, 'r', label='Validation acc')
plt.title('Training and validation accuracy')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.legend()
plt.show()

n=300  #Select the index of image to be loaded for testing
img = X_test[n]
plt.imshow(img)
input_img = np.expand_dims(img, axis=0) #Expand dims so the input is (num images, x, y, c)
print("The prediction for this image is: ", np.argmax(model.predict(input_img)))
print("The actual label for this image is: ", np.argmax(y_test[n]))

from sklearn.metrics import confusion_matrix
import seaborn as sns

y_pred = np.argmax(model.predict(X_test), axis=1)
cm=confusion_matrix(np.argmax(y_test, axis=1), y_pred)  
sns.heatmap(cm, annot=True)

#Identify all images classified as parasitized
parasited_image_idx = np.where(y_pred == 1)[0]

predicted_as_para=[]
for i in parasited_image_idx:
    par_img = X_test[i]
    #plt.imsave("results_classified_as_para/para_"+str(i)+".png", par_img)
    predicted_as_para.append(par_img)
    
predicted_as_para = np.array(predicted_as_para)   

```
        
        
        
        
        
        
        
        
        
        
        

### Standalone code to reproduce the issue

```shell
from sklearn.metrics import confusion_matrix
import seaborn as sns

y_pred = np.argmax(model.predict(X_test), axis=1)
cm=confusion_matrix(np.argmax(y_test, axis=1), y_pred)  
sns.heatmap(cm, annot=True)
```


### Relevant log output

```shell
Traceback (most recent call last):

  Cell In[2], line 4
    y_pred = np.argmax(model.predict(X_test), axis=1)

  File D:\anaconda3\envs\tensorflow-gpu\lib\site-packages\keras\utils\traceback_utils.py:70 in error_handler
    raise e.with_traceback(filtered_tb) from None

  File D:\anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\framework\constant_op.py:102 in convert_to_eager_tensor
    return ops.EagerTensor(value, ctx.device_name, dtype)

InternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run _EagerConst: Dst tensor is not initialized.


2023-04-23 01:00:06.362905: I tensorflow/stream_executor/cuda/cuda_driver.cc:733] failed to allocate 8.41G (9029127168 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-04-23 01:00:06.507445: I tensorflow/stream_executor/cuda/cuda_driver.cc:733] failed to allocate 8.41G (9029127168 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-04-23 01:00:16.685829: I tensorflow/stream_executor/cuda/cuda_driver.cc:733] failed to allocate 8.41G (9029127168 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-04-23 01:00:16.820275: I tensorflow/stream_executor/cuda/cuda_driver.cc:733] failed to allocate 8.41G (9029127168 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-04-23 01:00:16.820335: W tensorflow/core/common_runtime/bfc_allocator.cc:479] Allocator (GPU_0_bfc) ran out of memory trying to allocate 560.44MiB (rounded to 587661312)requested by op _EagerConst
If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation. 
Current allocation summary follows.
Current allocation summary follows.
2023-04-23 01:00:16.820350: I tensorflow/core/common_runtime/bfc_allocator.cc:1033] BFCAllocator dump for GPU_0_bfc
2023-04-23 01:00:16.820363: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Bin (256): 	Total Chunks: 45, Chunks in use: 44. 11.2KiB allocated for chunks. 11.0KiB in use in bin. 740B client-requested in use in bin.
2023-04-23 01:00:16.820380: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Bin (512): 	Total Chunks: 2, Chunks in use: 2. 1.0KiB allocated for chunks. 1.0KiB in use in bin. 1.0KiB client-requested in use in bin.
2023-04-23 01:00:16.820394: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Bin (1024): 	Total Chunks: 4, Chunks in use: 4. 4.2KiB allocated for chunks. 4.2KiB in use in bin. 4.0KiB client-requested in use in bin.
2023-04-23 01:00:16.820405: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Bin (2048): 	Total Chunks: 9, Chunks in use: 9. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 18.0KiB client-requested in use in bin.
2023-04-23 01:00:16.820416: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Bin (4096): 	Total Chunks: 4, Chunks in use: 4. 22.5KiB allocated for chunks. 22.5KiB in use in bin. 22.4KiB client-requested in use in bin.
2023-04-23 01:00:16.820426: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Bin (8192): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-04-23 01:00:16.820437: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Bin (16384): 	Total Chunks: 1, Chunks in use: 1. 30.5KiB allocated for chunks. 30.5KiB in use in bin. 30.5KiB client-requested in use in bin.
2023-04-23 01:00:16.820447: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Bin (32768): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-04-23 01:00:16.820456: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Bin (65536): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-04-23 01:00:16.820466: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Bin (131072): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-04-23 01:00:16.820477: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Bin (262144): 	Total Chunks: 2, Chunks in use: 2. 707.0KiB allocated for chunks. 707.0KiB in use in bin. 432.0KiB client-requested in use in bin.
2023-04-23 01:00:16.820488: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Bin (524288): 	Total Chunks: 2, Chunks in use: 2. 1.14MiB allocated for chunks. 1.14MiB in use in bin. 1.14MiB client-requested in use in bin.
2023-04-23 01:00:16.820499: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Bin (1048576): 	Total Chunks: 1, Chunks in use: 1. 1.97MiB allocated for chunks. 1.97MiB in use in bin. 1.12MiB client-requested in use in bin.
2023-04-23 01:00:16.820510: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Bin (2097152): 	Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2023-04-23 01:00:16.820520: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Bin (4194304): 	Total Chunks: 2, Chunks in use: 1. 10.06MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2023-04-23 01:00:16.820531: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Bin (8388608): 	Total Chunks: 8, Chunks in use: 8. 72.00MiB allocated for chunks. 72.00MiB in use in bin. 72.00MiB client-requested in use in bin.
2023-04-23 01:00:16.820541: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Bin (16777216): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-04-23 01:00:16.820551: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Bin (33554432): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-04-23 01:00:16.820560: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Bin (67108864): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-04-23 01:00:16.820572: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Bin (134217728): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-04-23 01:00:16.820584: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Bin (268435456): 	Total Chunks: 3, Chunks in use: 2. 3.22GiB allocated for chunks. 2.74GiB in use in bin. 2.74GiB client-requested in use in bin.
2023-04-23 01:00:16.820594: I tensorflow/core/common_runtime/bfc_allocator.cc:1056] Bin for 560.44MiB was 256.00MiB, Chunk State: 
2023-04-23 01:00:16.820607: I tensorflow/core/common_runtime/bfc_allocator.cc:1062]   Size: 497.67MiB | Requested Size: 9.00MiB | in_use: 0 | bin_num: 20, prev:   Size: 560.44MiB | Requested Size: 560.44MiB | in_use: 1 | bin_num: -1
2023-04-23 01:00:16.820615: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] Next region of size 3553784832
2023-04-23 01:00:16.820623: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa00000 of size 1280 next 1
2023-04-23 01:00:16.820630: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa00500 of size 256 next 2
2023-04-23 01:00:16.820637: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa00600 of size 256 next 3
2023-04-23 01:00:16.820644: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa00700 of size 256 next 5
2023-04-23 01:00:16.820651: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa00800 of size 256 next 6
2023-04-23 01:00:16.820658: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa00900 of size 256 next 4
2023-04-23 01:00:16.820665: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa00a00 of size 256 next 7
2023-04-23 01:00:16.820672: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa00b00 of size 256 next 12
2023-04-23 01:00:16.820679: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa00c00 of size 256 next 10
2023-04-23 01:00:16.820686: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa00d00 of size 256 next 11
2023-04-23 01:00:16.820693: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa00e00 of size 512 next 15
2023-04-23 01:00:16.820700: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa01000 of size 256 next 16
2023-04-23 01:00:16.820707: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa01100 of size 256 next 19
2023-04-23 01:00:16.820714: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa01200 of size 256 next 50
2023-04-23 01:00:16.820722: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa01300 of size 256 next 22
2023-04-23 01:00:16.820729: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa01400 of size 256 next 20
2023-04-23 01:00:16.820736: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa01500 of size 256 next 21
2023-04-23 01:00:16.820743: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa01600 of size 1024 next 25
2023-04-23 01:00:16.820750: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa01a00 of size 256 next 26
2023-04-23 01:00:16.820757: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa01b00 of size 256 next 29
2023-04-23 01:00:16.820764: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa01c00 of size 1024 next 32
2023-04-23 01:00:16.820771: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa02000 of size 256 next 56
2023-04-23 01:00:16.820777: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa02100 of size 256 next 57
2023-04-23 01:00:16.820784: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa02200 of size 256 next 58
2023-04-23 01:00:16.820791: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa02300 of size 256 next 35
2023-04-23 01:00:16.820799: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa02400 of size 256 next 30
2023-04-23 01:00:16.820806: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa02500 of size 256 next 31
2023-04-23 01:00:16.820813: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa02600 of size 2048 next 37
2023-04-23 01:00:16.820820: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa02e00 of size 256 next 38
2023-04-23 01:00:16.820827: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa02f00 of size 256 next 41
2023-04-23 01:00:16.820834: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa03000 of size 2048 next 44
2023-04-23 01:00:16.820841: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa03800 of size 2048 next 8
2023-04-23 01:00:16.820848: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa04000 of size 256 next 54
2023-04-23 01:00:16.820855: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa04100 of size 512 next 17
2023-04-23 01:00:16.820863: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa04300 of size 1024 next 28
2023-04-23 01:00:16.820870: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa04700 of size 2048 next 39
2023-04-23 01:00:16.820877: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa04f00 of size 2048 next 49
2023-04-23 01:00:16.820884: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa05700 of size 2048 next 51
2023-04-23 01:00:16.820891: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa05f00 of size 2048 next 65
2023-04-23 01:00:16.820898: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa06700 of size 3072 next 52
2023-04-23 01:00:16.820908: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa07300 of size 256 next 47
2023-04-23 01:00:16.820916: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa07400 of size 6912 next 42
2023-04-23 01:00:16.820923: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa08f00 of size 281600 next 14
2023-04-23 01:00:16.820930: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aa4db00 of size 442368 next 18
2023-04-23 01:00:16.820937: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50aab9b00 of size 2064384 next 23
2023-04-23 01:00:16.820944: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50acb1b00 of size 589824 next 13
2023-04-23 01:00:16.820951: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50ad41b00 of size 4718592 next 33
2023-04-23 01:00:16.820957: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50b1c1b00 of size 256 next 59
2023-04-23 01:00:16.820964: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50b1c1c00 of size 256 next 60
2023-04-23 01:00:16.820971: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50b1c1d00 of size 256 next 61
2023-04-23 01:00:16.820978: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50b1c1e00 of size 256 next 62
2023-04-23 01:00:16.820985: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50b1c1f00 of size 3072 next 24
2023-04-23 01:00:16.820992: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50b1c2b00 of size 4096 next 48
2023-04-23 01:00:16.821000: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50b1c3b00 of size 31232 next 55
2023-04-23 01:00:16.821007: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50b1cb500 of size 4096 next 67
2023-04-23 01:00:16.821014: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50b1cc500 of size 256 next 68
2023-04-23 01:00:16.821021: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50b1cc600 of size 256 next 69
2023-04-23 01:00:16.821028: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50b1cc700 of size 256 next 70
2023-04-23 01:00:16.821034: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50b1cc800 of size 256 next 71
2023-04-23 01:00:16.821042: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50b1cc900 of size 256 next 72
2023-04-23 01:00:16.821050: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50b1cca00 of size 256 next 73
2023-04-23 01:00:16.821057: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50b1ccb00 of size 256 next 74
2023-04-23 01:00:16.821064: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50b1ccc00 of size 256 next 75
2023-04-23 01:00:16.821070: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50b1ccd00 of size 256 next 76
2023-04-23 01:00:16.821077: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50b1cce00 of size 256 next 81
2023-04-23 01:00:16.821084: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50b1ccf00 of size 256 next 78
2023-04-23 01:00:16.821091: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] Free  at 50b1cd000 of size 256 next 85
2023-04-23 01:00:16.821098: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50b1cd100 of size 256 next 96
2023-04-23 01:00:16.821104: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50b1cd200 of size 256 next 86
2023-04-23 01:00:16.821111: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50b1cd300 of size 7936 next 93
2023-04-23 01:00:16.821118: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50b1cf200 of size 602112 next 89
2023-04-23 01:00:16.821125: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] Free  at 50b262200 of size 5830912 next 36
2023-04-23 01:00:16.821132: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50b7f1b00 of size 2359296 next 27
2023-04-23 01:00:16.821140: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50ba31b00 of size 2359296 next 40
2023-04-23 01:00:16.821147: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50bc71b00 of size 9437184 next 34
2023-04-23 01:00:16.821154: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50c571b00 of size 9437184 next 46
2023-04-23 01:00:16.821161: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50ce71b00 of size 9437184 next 45
2023-04-23 01:00:16.821168: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50d771b00 of size 9437184 next 9
2023-04-23 01:00:16.821174: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50e071b00 of size 9437184 next 43
2023-04-23 01:00:16.821181: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 50e971b00 of size 2349441024 next 53
2023-04-23 01:00:16.821188: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 59aa0bb00 of size 9437184 next 63
2023-04-23 01:00:16.821195: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 59b30bb00 of size 9437184 next 64
2023-04-23 01:00:16.821202: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 59bc0bb00 of size 9437184 next 66
2023-04-23 01:00:16.821209: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] InUse at 59c50bb00 of size 587661312 next 84
2023-04-23 01:00:16.821216: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] Free  at 5bf57bb00 of size 521844992 next 18446744073709551615
2023-04-23 01:00:16.821223: I tensorflow/core/common_runtime/bfc_allocator.cc:1094]      Summary of in-use Chunks by size: 
2023-04-23 01:00:16.821232: I tensorflow/core/common_runtime/bfc_allocator.cc:1097] 44 Chunks of size 256 totalling 11.0KiB
2023-04-23 01:00:16.821241: I tensorflow/core/common_runtime/bfc_allocator.cc:1097] 2 Chunks of size 512 totalling 1.0KiB
2023-04-23 01:00:16.821249: I tensorflow/core/common_runtime/bfc_allocator.cc:1097] 3 Chunks of size 1024 totalling 3.0KiB
2023-04-23 01:00:16.821258: I tensorflow/core/common_runtime/bfc_allocator.cc:1097] 1 Chunks of size 1280 totalling 1.2KiB
2023-04-23 01:00:16.821267: I tensorflow/core/common_runtime/bfc_allocator.cc:1097] 7 Chunks of size 2048 totalling 14.0KiB
2023-04-23 01:00:16.821277: I tensorflow/core/common_runtime/bfc_allocator.cc:1097] 2 Chunks of size 3072 totalling 6.0KiB
2023-04-23 01:00:16.821286: I tensorflow/core/common_runtime/bfc_allocator.cc:1097] 2 Chunks of size 4096 totalling 8.0KiB
2023-04-23 01:00:16.821294: I tensorflow/core/common_runtime/bfc_allocator.cc:1097] 1 Chunks of size 6912 totalling 6.8KiB
2023-04-23 01:00:16.821303: I tensorflow/core/common_runtime/bfc_allocator.cc:1097] 1 Chunks of size 7936 totalling 7.8KiB
2023-04-23 01:00:16.821313: I tensorflow/core/common_runtime/bfc_allocator.cc:1097] 1 Chunks of size 31232 totalling 30.5KiB
2023-04-23 01:00:16.821322: I tensorflow/core/common_runtime/bfc_allocator.cc:1097] 1 Chunks of size 281600 totalling 275.0KiB
2023-04-23 01:00:16.821330: I tensorflow/core/common_runtime/bfc_allocator.cc:1097] 1 Chunks of size 442368 totalling 432.0KiB
2023-04-23 01:00:16.821339: I tensorflow/core/common_runtime/bfc_allocator.cc:1097] 1 Chunks of size 589824 totalling 576.0KiB
2023-04-23 01:00:16.821347: I tensorflow/core/common_runtime/bfc_allocator.cc:1097] 1 Chunks of size 602112 totalling 588.0KiB
2023-04-23 01:00:16.821356: I tensorflow/core/common_runtime/bfc_allocator.cc:1097] 1 Chunks of size 2064384 totalling 1.97MiB
2023-04-23 01:00:16.821365: I tensorflow/core/common_runtime/bfc_allocator.cc:1097] 2 Chunks of size 2359296 totalling 4.50MiB
2023-04-23 01:00:16.821373: I tensorflow/core/common_runtime/bfc_allocator.cc:1097] 1 Chunks of size 4718592 totalling 4.50MiB
2023-04-23 01:00:16.821382: I tensorflow/core/common_runtime/bfc_allocator.cc:1097] 8 Chunks of size 9437184 totalling 72.00MiB
2023-04-23 01:00:16.821390: I tensorflow/core/common_runtime/bfc_allocator.cc:1097] 1 Chunks of size 587661312 totalling 560.44MiB
2023-04-23 01:00:16.821399: I tensorflow/core/common_runtime/bfc_allocator.cc:1097] 1 Chunks of size 2349441024 totalling 2.19GiB
2023-04-23 01:00:16.821407: I tensorflow/core/common_runtime/bfc_allocator.cc:1101] Sum Total of in-use chunks: 2.82GiB
2023-04-23 01:00:16.821416: I tensorflow/core/common_runtime/bfc_allocator.cc:1103] total_region_allocated_bytes_: 3553784832 memory_limit_: 12582912000 available bytes: 9029127168 curr_region_allocation_bytes_: 25165824000
2023-04-23 01:00:16.821428: I tensorflow/core/common_runtime/bfc_allocator.cc:1109] Stats: 
Limit:                     12582912000
InUse:                      3026108672
MaxInUse:                   3553784320
NumAllocs:                      299589
MaxAllocSize:               2349441024
Reserved:                            0
PeakReserved:                        0
LargestFreeBlock:                    0

2023-04-23 01:00:16.821441: W tensorflow/core/common_runtime/bfc_allocator.cc:491] **************************************************************************************______________
```
</details>