model.fit() occur "Cudnn graph failed to build: UNKNOWN: CUDNN_STATUS_BAD_PARAM"
Hi @ipmnhmyang ,

Normally this type of error happens due to cuDNN/CUDA mismatch and also  wrt TF version. But from your inputs you are confirming that only tested configurations. We need to check your environment details.Please confirm below.

1. Are you using pre built binaries provided by tensorflow or used build from source ?
2. If using pre built binaries ensure it is installed by pip only not conda
3. If used build from source please confirm whether it was built with respective CUDA and cuDNN.

Please confirm whether you have followed all the instructions as per official documentation mentioned for [pip](https://www.tensorflow.org/install/pip) or build from [source](https://www.tensorflow.org/install/source#ubuntu).

Provided code snippet has dependency with image directory. Please provide a reproducible code snippet with an accessible dataset.

Thanks!

 Please also have a look of similar issue at #41060 for any pointers there. Thanks! @SuryanarayanaY Hi, thank you for reply.

I've confirmed configuration and tested one more time.
After configureation following https://www.tensorflow.org/install/pip, it is working properly.

Thanks a lot. Problem solved. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61403">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61403">No</a>
 @ipmnhmyang ,

For more clarity for us and community, could you please confirm both RTX 3090 Ti & RTX 4090 are able to detectable using CUDA-11.8 and cuDNN-8.6 for TF2.13 version which is tested configuration.

Thanks!
### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

source

### TensorFlow version

2.11, 2.12, 2.13

### Custom code

Yes

### OS platform and distribution

Linux Ubuntu 22.04

### Mobile device

_No response_

### Python version

3.10

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

11.8 / 8.6 & 11.8 / 8.9.2

### GPU model and memory

RTX 3090 Ti & RTX 4090

### Current behavior?

This is first time experience to have such error message.

When I try
"model.fit()"
server stops with error message below

Tried cuDNN version 8.6 (as [tensorflow.org](https://www.tensorflow.org/install/pip) ) and 8.9.2 (lateset for CUDA 11.8)
Both have problem.

How can I solve the issue?
Thanks!

### Standalone code to reproduce the issue

```shell
gpu_id = "2" # 0 or 1
import os
os.environ["CUDA_VISIBLE_DEVICES"] = gpu_id

import tensorflow as tf
import tensorflow.keras as keras
import tensorflow.keras.layers as layers
import tensorflow.keras.models as models

size_y = 256
size_x = 256

#--- load dataset
dic_path = './seg_dataset/train/dic'
msk_path = './seg_dataset/train/msk'

seed = 1004 # random number in your mind

dic_datagen = keras.preprocessing.image.ImageDataGenerator(
        rescale=1./255,
        validation_split=0.2
    )
msk_datagen = keras.preprocessing.image.ImageDataGenerator(
        rescale=1./255,
        validation_split=0.2
    )

dic_train = \
    dic_datagen.flow_from_directory( 
        dic_path,
        target_size=(size_y, size_x),
        class_mode=None, 
        seed=seed,
        subset='training'
    )

msk_train = \
    msk_datagen.flow_from_directory( 
        msk_path,
        target_size=(size_y, size_x),
        class_mode=None,
        color_mode='grayscale',
        seed=seed,
        subset='training'
    )

dic_valid = \
    dic_datagen.flow_from_directory( 
        dic_path,
        target_size=(size_y, size_x),
        class_mode=None, 
        seed=seed,
        subset='validation'
    )
msk_valid = \
    msk_datagen.flow_from_directory( 
        msk_path,
        target_size=(size_y, size_x),
        class_mode=None,
        color_mode='grayscale',
        seed=seed,
        subset='validation'
    )

train_ds = zip(dic_train, msk_train)
valid_ds = zip(dic_valid, msk_valid)

f = [16, 32, 64, 128, 256]

kernel_size=(3,3)
padding='same'
strides=1


# number of filters at each level
inputs = layers.Input((size_y, size_x, 1))
p0 = inputs
# downblock 1
x  = layers.Conv2D(16,  kernel_size, padding=padding, strides=strides, activation="relu")(p0)
c1 = layers.Conv2D(16, kernel_size, padding=padding, strides=strides, activation="relu")(x)
x  = layers.MaxPool2D((2, 2), (2, 2))(c1)
# downblock 2
x  = layers.Conv2D(32,  kernel_size, padding=padding, strides=strides, activation="relu")(x)
c2 = layers.Conv2D(32, kernel_size, padding=padding, strides=strides, activation="relu")(x)
x  = layers.MaxPool2D((2, 2), (2, 2))(c2)
# downblock 3
x  = layers.Conv2D(64,  kernel_size, padding=padding, strides=strides, activation="relu")(x)
c3 = layers.Conv2D(64, kernel_size, padding=padding, strides=strides, activation="relu")(x)
x  = layers.MaxPool2D((2, 2), (2, 2))(c3)
# downblock 4
x  = layers.Conv2D(128,  kernel_size, padding=padding, strides=strides, activation="relu")(x)
c4 = layers.Conv2D(128, kernel_size, padding=padding, strides=strides, activation="relu")(x)
x  = layers.MaxPool2D((2, 2), (2, 2))(c4)
# bottle neck
x  = layers.Conv2D(256, kernel_size, padding=padding, strides=strides, activation="relu")(x)
x  = layers.Conv2D(256, kernel_size, padding=padding, strides=strides, activation="relu")(x)
# up block 1
x  = layers.UpSampling2D((2, 2))(x)
concat = layers.Concatenate()([x, c4])
x  = layers.Conv2D(128, kernel_size, padding=padding, strides=strides, activation="relu")(concat)
x  = layers.Conv2D(128, kernel_size, padding=padding, strides=strides, activation="relu")(x)
# up block 1
x  = layers.UpSampling2D((2, 2))(x)
concat = layers.Concatenate()([x, c3])
x  = layers.Conv2D(64, kernel_size, padding=padding, strides=strides, activation="relu")(concat)
x  = layers.Conv2D(64, kernel_size, padding=padding, strides=strides, activation="relu")(x)
# up block 1
x  = layers.UpSampling2D((2, 2))(x)
concat = layers.Concatenate()([x, c2])
x  = layers.Conv2D(32, kernel_size, padding=padding, strides=strides, activation="relu")(concat)
x  = layers.Conv2D(32, kernel_size, padding=padding, strides=strides, activation="relu")(x)
# up block 1
x  = layers.UpSampling2D((2, 2))(x)
concat = layers.Concatenate()([x, c1])
x  = layers.Conv2D(16, kernel_size, padding=padding, strides=strides, activation="relu")(concat)
x  = layers.Conv2D(16, kernel_size, padding=padding, strides=strides, activation="relu")(x)

# last convolution 1x1
outputs = layers.Conv2D(1, (1, 1), padding="same", activation="sigmoid")(x)
model = models.Model(inputs, outputs)



model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

path_checkpoint = './seg_checkpoint'
os.makedirs(path_checkpoint,exist_ok=True)

model_checkpointer = keras.callbacks.ModelCheckpoint(
    filepath = path_checkpoint,
    save_weights_only=True,
    monitor='val_loss',
    mode='min',
    save_best_only=True,
    verbose = 1
)

#--- additional
callbacks = [
        model_checkpointer,
        keras.callbacks.EarlyStopping(
            patience=50*3,
            monitor='val_loss',
            mode='min',
            verbose=1
            ),
]



#--- train start
EPOCH = 10

history = model.fit(
        train_ds,
        validation_data=valid_ds,
        validation_steps=15, 
        # Total number of steps (batches of samples) 
        # to draw before stopping when performing validation at the end of every epoch.
        batch_size=16,
        steps_per_epoch=50,
        epochs=EPOCH,
        callbacks=callbacks
    )
```


### Relevant log output

```shell
2023-07-26 14:46:27.667380: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:8942] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-07-26 14:46:27.667411: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-07-26 14:46:27.667426: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-07-26 14:46:27.671343: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-07-26 14:46:28.183018: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
WARNING:tensorflow:From /home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/tensorflow/python/ops/distributions/distribution.py:259: ReparameterizationType.__init__ (from tensorflow.python.ops.distributions.distribution) is deprecated and will be removed after 2019-01-01.
Instructions for updating:
The TensorFlow Distributions library has moved to TensorFlow Probability (https://github.com/tensorflow/probability). You should update all references to use `tfp.distributions` instead of `tf.distributions`.
WARNING:tensorflow:From /home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/tensorflow/python/ops/distributions/bernoulli.py:165: RegisterKL.__init__ (from tensorflow.python.ops.distributions.kullback_leibler) is deprecated and will be removed after 2019-01-01.
Instructions for updating:
The TensorFlow Distributions library has moved to TensorFlow Probability (https://github.com/tensorflow/probability). You should update all references to use `tfp.distributions` instead of `tf.distributions`.
2023-07-26 14:46:28.727203: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-07-26 14:46:28.741660: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-07-26 14:46:28.741864: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-07-26 14:46:28.807551: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-07-26 14:46:28.807748: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-07-26 14:46:28.807913: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-07-26 14:46:28.808057: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1884] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 22168 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 3090 Ti, pci bus id: 0000:41:00.0, compute capability: 8.6
2023-07-26 14:46:28.809796: I tensorflow/core/common_runtime/direct_session.cc:380] Device mapping:
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: NVIDIA GeForce RTX 3090 Ti, pci bus id: 0000:41:00.0, compute capability: 8.6

Found 40000 images belonging to 1 classes.
Found 40000 images belonging to 1 classes.
Found 10000 images belonging to 1 classes.
Found 10000 images belonging to 1 classes.
2023-07-26 14:46:30.293048: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-07-26 14:46:30.293252: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-07-26 14:46:30.293411: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-07-26 14:46:30.293658: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-07-26 14:46:30.293824: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-07-26 14:46:30.293974: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-07-26 14:46:30.294158: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-07-26 14:46:30.294315: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:894] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-07-26 14:46:30.294450: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1884] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 22168 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 3090 Ti, pci bus id: 0000:41:00.0, compute capability: 8.6
Epoch 1/10
2023-07-26 14:46:31.666051: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:440] Loaded cuDNN version 8600
2023-07-26 14:46:31.674917: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at conv_ops_fused_impl.h:625 : INTERNAL: Cudnn graph failed to build: UNKNOWN: CUDNN_STATUS_BAD_PARAM
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(4340): 'conv_op' CUDNN_BACKEND_OPERATION: cudnnFinalize Failed
Traceback (most recent call last):
  File "/home/bootcamp/train_unet.py", line 161, in <module>
    history = model.fit(
  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/tensorflow/python/eager/execute.py", line 53, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InternalError: Graph execution error:

Detected at node model/conv2d/Relu defined at (most recent call last):
  File "/home/bootcamp/train_unet.py", line 161, in <module>
    history = model.fit(

  File "/home/bootcamp/train_unet.py", line 161, in <module>
    history = model.fit(

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/train_unet.py", line 161, in <module>
    history = model.fit(

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1783, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/bootcamp/train_unet.py", line 161, in <module>
    history = model.fit(

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1783, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1377, in train_function
    return step_function(self, iterator)

  File "/home/bootcamp/train_unet.py", line 161, in <module>
    history = model.fit(

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1783, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1377, in train_function
    return step_function(self, iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1360, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/bootcamp/train_unet.py", line 161, in <module>
    history = model.fit(

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1783, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1377, in train_function
    return step_function(self, iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1360, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1349, in run_step
    outputs = model.train_step(data)

  File "/home/bootcamp/train_unet.py", line 161, in <module>
    history = model.fit(

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1783, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1377, in train_function
    return step_function(self, iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1360, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1349, in run_step
    outputs = model.train_step(data)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1126, in train_step
    y_pred = self(x, training=True)

  File "/home/bootcamp/train_unet.py", line 161, in <module>
    history = model.fit(

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1783, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1377, in train_function
    return step_function(self, iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1360, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1349, in run_step
    outputs = model.train_step(data)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1126, in train_step
    y_pred = self(x, training=True)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/train_unet.py", line 161, in <module>
    history = model.fit(

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1783, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1377, in train_function
    return step_function(self, iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1360, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1349, in run_step
    outputs = model.train_step(data)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1126, in train_step
    y_pred = self(x, training=True)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 589, in __call__
    return super().__call__(*args, **kwargs)

  File "/home/bootcamp/train_unet.py", line 161, in <module>
    history = model.fit(

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1783, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1377, in train_function
    return step_function(self, iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1360, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1349, in run_step
    outputs = model.train_step(data)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1126, in train_step
    y_pred = self(x, training=True)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 589, in __call__
    return super().__call__(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/train_unet.py", line 161, in <module>
    history = model.fit(

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1783, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1377, in train_function
    return step_function(self, iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1360, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1349, in run_step
    outputs = model.train_step(data)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1126, in train_step
    y_pred = self(x, training=True)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 589, in __call__
    return super().__call__(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1149, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/bootcamp/train_unet.py", line 161, in <module>
    history = model.fit(

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1783, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1377, in train_function
    return step_function(self, iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1360, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1349, in run_step
    outputs = model.train_step(data)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1126, in train_step
    y_pred = self(x, training=True)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 589, in __call__
    return super().__call__(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1149, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/train_unet.py", line 161, in <module>
    history = model.fit(

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1783, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1377, in train_function
    return step_function(self, iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1360, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1349, in run_step
    outputs = model.train_step(data)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1126, in train_step
    y_pred = self(x, training=True)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 589, in __call__
    return super().__call__(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1149, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/functional.py", line 515, in call
    return self._run_internal_graph(inputs, training=training, mask=mask)

  File "/home/bootcamp/train_unet.py", line 161, in <module>
    history = model.fit(

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1783, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1377, in train_function
    return step_function(self, iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1360, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1349, in run_step
    outputs = model.train_step(data)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1126, in train_step
    y_pred = self(x, training=True)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 589, in __call__
    return super().__call__(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1149, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/functional.py", line 515, in call
    return self._run_internal_graph(inputs, training=training, mask=mask)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/functional.py", line 672, in _run_internal_graph
    outputs = node.layer(*args, **kwargs)

  File "/home/bootcamp/train_unet.py", line 161, in <module>
    history = model.fit(

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1783, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1377, in train_function
    return step_function(self, iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1360, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1349, in run_step
    outputs = model.train_step(data)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1126, in train_step
    y_pred = self(x, training=True)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 589, in __call__
    return super().__call__(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1149, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/functional.py", line 515, in call
    return self._run_internal_graph(inputs, training=training, mask=mask)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/functional.py", line 672, in _run_internal_graph
    outputs = node.layer(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/train_unet.py", line 161, in <module>
    history = model.fit(

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1783, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1377, in train_function
    return step_function(self, iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1360, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1349, in run_step
    outputs = model.train_step(data)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1126, in train_step
    y_pred = self(x, training=True)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 589, in __call__
    return super().__call__(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1149, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/functional.py", line 515, in call
    return self._run_internal_graph(inputs, training=training, mask=mask)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/functional.py", line 672, in _run_internal_graph
    outputs = node.layer(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1149, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/bootcamp/train_unet.py", line 161, in <module>
    history = model.fit(

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1783, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1377, in train_function
    return step_function(self, iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1360, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1349, in run_step
    outputs = model.train_step(data)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1126, in train_step
    y_pred = self(x, training=True)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 589, in __call__
    return super().__call__(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1149, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/functional.py", line 515, in call
    return self._run_internal_graph(inputs, training=training, mask=mask)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/functional.py", line 672, in _run_internal_graph
    outputs = node.layer(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1149, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/train_unet.py", line 161, in <module>
    history = model.fit(

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1783, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1377, in train_function
    return step_function(self, iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1360, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1349, in run_step
    outputs = model.train_step(data)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1126, in train_step
    y_pred = self(x, training=True)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 589, in __call__
    return super().__call__(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1149, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/functional.py", line 515, in call
    return self._run_internal_graph(inputs, training=training, mask=mask)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/functional.py", line 672, in _run_internal_graph
    outputs = node.layer(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1149, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/layers/convolutional/base_conv.py", line 321, in call
    return self.activation(outputs)

  File "/home/bootcamp/train_unet.py", line 161, in <module>
    history = model.fit(

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1783, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1377, in train_function
    return step_function(self, iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1360, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1349, in run_step
    outputs = model.train_step(data)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1126, in train_step
    y_pred = self(x, training=True)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 589, in __call__
    return super().__call__(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1149, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/functional.py", line 515, in call
    return self._run_internal_graph(inputs, training=training, mask=mask)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/functional.py", line 672, in _run_internal_graph
    outputs = node.layer(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1149, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/layers/convolutional/base_conv.py", line 321, in call
    return self.activation(outputs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/activations.py", line 306, in relu
    return backend.relu(

  File "/home/bootcamp/train_unet.py", line 161, in <module>
    history = model.fit(

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1783, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1377, in train_function
    return step_function(self, iterator)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1360, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1349, in run_step
    outputs = model.train_step(data)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 1126, in train_step
    y_pred = self(x, training=True)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/training.py", line 589, in __call__
    return super().__call__(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1149, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/functional.py", line 515, in call
    return self._run_internal_graph(inputs, training=training, mask=mask)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/functional.py", line 672, in _run_internal_graph
    outputs = node.layer(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1149, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/layers/convolutional/base_conv.py", line 321, in call
    return self.activation(outputs)

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/activations.py", line 306, in relu
    return backend.relu(

  File "/home/bootcamp/miniconda3/envs/tf/lib/python3.10/site-packages/keras/src/backend.py", line 5397, in relu
    x = tf.nn.relu(x)

Cudnn graph failed to build: UNKNOWN: CUDNN_STATUS_BAD_PARAM
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(4340): 'conv_op' CUDNN_BACKEND_OPERATION: cudnnFinalize Failed
         [[{{node model/conv2d/Relu}}]] [Op:__inference_train_function_4359]
```
