TensorFlow can't communicate with GPU. "None of the algorithms provided by cuDNN frontend heuristics worked; trying fallback algorithms." "No algorithm worked!"
On the nightly version the error output is a bit different, maybe this is helpful:

```
(tf-nightly) [lnapon@rhel ancient_ai]$ python3 lrnkeras/mnist_conv.py 
2023-05-25 19:41:31.002823: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:7704] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-05-25 19:41:31.002868: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-05-25 19:41:31.002884: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1520] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-05-25 19:41:31.847714: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-05-25 19:41:32.672538: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-25 19:41:32.688679: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-25 19:41:32.688911: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-25 19:41:32.689738: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-25 19:41:32.689894: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-25 19:41:32.690034: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-25 19:41:33.135930: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-25 19:41:33.136133: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-25 19:41:33.136290: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-25 19:41:33.136425: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1318 MB memory:  -> device: 0, name: Quadro K620, pci bus id: 0000:01:00.0, compute capability: 5.0
Epoch 1/5
2023-05-25 19:41:34.396173: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:434] Loaded cuDNN version 8600
2023-05-25 19:41:34.653261: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:190] failed to create cublas handle: the resource allocation failed
2023-05-25 19:41:34.653298: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:193] Failure to initialize cublas may be due to OOM (cublas needs some free memory when you initialize it, and your deep-learning framework may have preallocated more than its fair share), or may be because this binary was not built with support for the GPU in your machine.
2023-05-25 19:41:34.653625: W tensorflow/core/kernels/conv_ops_gpu.cc:144] None of the algorithms provided by cuDNN frontend heuristics worked; trying fallback algorithms.  Conv: batch: 1
in_depths: 1
out_depths: 4
in: 28
in: 28
data_format: 1
filter: 3
filter: 3
filter: 1
dilation: 1
dilation: 1
stride: 1
stride: 1
padding: 0
padding: 0
dtype: DT_FLOAT
group_count: 1
device_identifier: "sm_5.0 with 2099118080B RAM, 3 cores, 1124000KHz clock, 900000KHz mem clock, 2097152B L2$"
fusion {
  activation_mode: kRelu
  conv_scale: 1
}
version: 3

2023-05-25 19:41:34.660525: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:190] failed to create cublas handle: the resource allocation failed
2023-05-25 19:41:34.660542: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:193] Failure to initialize cublas may be due to OOM (cublas needs some free memory when you initialize it, and your deep-learning framework may have preallocated more than its fair share), or may be because this binary was not built with support for the GPU in your machine.
2023-05-25 19:41:34.660782: W tensorflow/core/framework/op_kernel.cc:1828] OP_REQUIRES failed at conv_ops_fused_impl.h:625 : NOT_FOUND: No algorithm worked!  Error messages:
  Profiling failure on CUDNN engine eng11{}: UNKNOWN: CUDNN_STATUS_ALLOC_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5333): 'status'
  Profiling failure on CUDNN engine eng0{}: UNKNOWN: CUDNN_STATUS_ALLOC_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5333): 'status'
Traceback (most recent call last):
  File "/home/lnapon/repos/ancient_ai/lrnkeras/mnist_conv.py", line 25, in <module>
    model.fit(train_images, trains_labels, epochs=5, batch_size=1)
  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/tensorflow/python/eager/execute.py", line 53, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.NotFoundError: Graph execution error:

Detected at node sequential/conv2d/Relu defined at (most recent call last):
  File "/home/lnapon/repos/ancient_ai/lrnkeras/mnist_conv.py", line 25, in <module>
    model.fit(train_images, trains_labels, epochs=5, batch_size=1)

  File "/home/lnapon/repos/ancient_ai/lrnkeras/mnist_conv.py", line 25, in <module>
    model.fit(train_images, trains_labels, epochs=5, batch_size=1)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/repos/ancient_ai/lrnkeras/mnist_conv.py", line 25, in <module>
    model.fit(train_images, trains_labels, epochs=5, batch_size=1)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1768, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/lnapon/repos/ancient_ai/lrnkeras/mnist_conv.py", line 25, in <module>
    model.fit(train_images, trains_labels, epochs=5, batch_size=1)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1768, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1364, in train_function
    return step_function(self, iterator)

  File "/home/lnapon/repos/ancient_ai/lrnkeras/mnist_conv.py", line 25, in <module>
    model.fit(train_images, trains_labels, epochs=5, batch_size=1)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1768, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1364, in train_function
    return step_function(self, iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1348, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/lnapon/repos/ancient_ai/lrnkeras/mnist_conv.py", line 25, in <module>
    model.fit(train_images, trains_labels, epochs=5, batch_size=1)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1768, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1364, in train_function
    return step_function(self, iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1348, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1329, in run_step
    outputs = model.train_step(data)

  File "/home/lnapon/repos/ancient_ai/lrnkeras/mnist_conv.py", line 25, in <module>
    model.fit(train_images, trains_labels, epochs=5, batch_size=1)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1768, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1364, in train_function
    return step_function(self, iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1348, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1329, in run_step
    outputs = model.train_step(data)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1106, in train_step
    y_pred = self(x, training=True)

  File "/home/lnapon/repos/ancient_ai/lrnkeras/mnist_conv.py", line 25, in <module>
    model.fit(train_images, trains_labels, epochs=5, batch_size=1)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1768, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1364, in train_function
    return step_function(self, iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1348, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1329, in run_step
    outputs = model.train_step(data)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1106, in train_step
    y_pred = self(x, training=True)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/repos/ancient_ai/lrnkeras/mnist_conv.py", line 25, in <module>
    model.fit(train_images, trains_labels, epochs=5, batch_size=1)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1768, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1364, in train_function
    return step_function(self, iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1348, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1329, in run_step
    outputs = model.train_step(data)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1106, in train_step
    y_pred = self(x, training=True)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 588, in __call__
    return super().__call__(*args, **kwargs)

  File "/home/lnapon/repos/ancient_ai/lrnkeras/mnist_conv.py", line 25, in <module>
    model.fit(train_images, trains_labels, epochs=5, batch_size=1)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1768, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1364, in train_function
    return step_function(self, iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1348, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1329, in run_step
    outputs = model.train_step(data)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1106, in train_step
    y_pred = self(x, training=True)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 588, in __call__
    return super().__call__(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/repos/ancient_ai/lrnkeras/mnist_conv.py", line 25, in <module>
    model.fit(train_images, trains_labels, epochs=5, batch_size=1)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1768, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1364, in train_function
    return step_function(self, iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1348, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1329, in run_step
    outputs = model.train_step(data)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1106, in train_step
    y_pred = self(x, training=True)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 588, in __call__
    return super().__call__(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1150, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/lnapon/repos/ancient_ai/lrnkeras/mnist_conv.py", line 25, in <module>
    model.fit(train_images, trains_labels, epochs=5, batch_size=1)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1768, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1364, in train_function
    return step_function(self, iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1348, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1329, in run_step
    outputs = model.train_step(data)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1106, in train_step
    y_pred = self(x, training=True)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 588, in __call__
    return super().__call__(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1150, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/repos/ancient_ai/lrnkeras/mnist_conv.py", line 25, in <module>
    model.fit(train_images, trains_labels, epochs=5, batch_size=1)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1768, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1364, in train_function
    return step_function(self, iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1348, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1329, in run_step
    outputs = model.train_step(data)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1106, in train_step
    y_pred = self(x, training=True)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 588, in __call__
    return super().__call__(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1150, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/sequential.py", line 404, in call
    return super().call(inputs, training=training, mask=mask)

  File "/home/lnapon/repos/ancient_ai/lrnkeras/mnist_conv.py", line 25, in <module>
    model.fit(train_images, trains_labels, epochs=5, batch_size=1)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1768, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1364, in train_function
    return step_function(self, iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1348, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1329, in run_step
    outputs = model.train_step(data)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1106, in train_step
    y_pred = self(x, training=True)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 588, in __call__
    return super().__call__(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1150, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/sequential.py", line 404, in call
    return super().call(inputs, training=training, mask=mask)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/functional.py", line 512, in call
    return self._run_internal_graph(inputs, training=training, mask=mask)

  File "/home/lnapon/repos/ancient_ai/lrnkeras/mnist_conv.py", line 25, in <module>
    model.fit(train_images, trains_labels, epochs=5, batch_size=1)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1768, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1364, in train_function
    return step_function(self, iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1348, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1329, in run_step
    outputs = model.train_step(data)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1106, in train_step
    y_pred = self(x, training=True)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 588, in __call__
    return super().__call__(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1150, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/sequential.py", line 404, in call
    return super().call(inputs, training=training, mask=mask)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/functional.py", line 512, in call
    return self._run_internal_graph(inputs, training=training, mask=mask)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/functional.py", line 669, in _run_internal_graph
    outputs = node.layer(*args, **kwargs)

  File "/home/lnapon/repos/ancient_ai/lrnkeras/mnist_conv.py", line 25, in <module>
    model.fit(train_images, trains_labels, epochs=5, batch_size=1)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1768, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1364, in train_function
    return step_function(self, iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1348, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1329, in run_step
    outputs = model.train_step(data)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1106, in train_step
    y_pred = self(x, training=True)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 588, in __call__
    return super().__call__(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1150, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/sequential.py", line 404, in call
    return super().call(inputs, training=training, mask=mask)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/functional.py", line 512, in call
    return self._run_internal_graph(inputs, training=training, mask=mask)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/functional.py", line 669, in _run_internal_graph
    outputs = node.layer(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/repos/ancient_ai/lrnkeras/mnist_conv.py", line 25, in <module>
    model.fit(train_images, trains_labels, epochs=5, batch_size=1)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1768, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1364, in train_function
    return step_function(self, iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1348, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1329, in run_step
    outputs = model.train_step(data)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1106, in train_step
    y_pred = self(x, training=True)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 588, in __call__
    return super().__call__(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1150, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/sequential.py", line 404, in call
    return super().call(inputs, training=training, mask=mask)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/functional.py", line 512, in call
    return self._run_internal_graph(inputs, training=training, mask=mask)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/functional.py", line 669, in _run_internal_graph
    outputs = node.layer(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1150, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/lnapon/repos/ancient_ai/lrnkeras/mnist_conv.py", line 25, in <module>
    model.fit(train_images, trains_labels, epochs=5, batch_size=1)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1768, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1364, in train_function
    return step_function(self, iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1348, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1329, in run_step
    outputs = model.train_step(data)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1106, in train_step
    y_pred = self(x, training=True)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 588, in __call__
    return super().__call__(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1150, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/sequential.py", line 404, in call
    return super().call(inputs, training=training, mask=mask)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/functional.py", line 512, in call
    return self._run_internal_graph(inputs, training=training, mask=mask)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/functional.py", line 669, in _run_internal_graph
    outputs = node.layer(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1150, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/repos/ancient_ai/lrnkeras/mnist_conv.py", line 25, in <module>
    model.fit(train_images, trains_labels, epochs=5, batch_size=1)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1768, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1364, in train_function
    return step_function(self, iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1348, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1329, in run_step
    outputs = model.train_step(data)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1106, in train_step
    y_pred = self(x, training=True)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 588, in __call__
    return super().__call__(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1150, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/sequential.py", line 404, in call
    return super().call(inputs, training=training, mask=mask)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/functional.py", line 512, in call
    return self._run_internal_graph(inputs, training=training, mask=mask)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/functional.py", line 669, in _run_internal_graph
    outputs = node.layer(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1150, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/layers/convolutional/base_conv.py", line 321, in call
    return self.activation(outputs)

  File "/home/lnapon/repos/ancient_ai/lrnkeras/mnist_conv.py", line 25, in <module>
    model.fit(train_images, trains_labels, epochs=5, batch_size=1)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1768, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1364, in train_function
    return step_function(self, iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1348, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1329, in run_step
    outputs = model.train_step(data)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1106, in train_step
    y_pred = self(x, training=True)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 588, in __call__
    return super().__call__(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1150, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/sequential.py", line 404, in call
    return super().call(inputs, training=training, mask=mask)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/functional.py", line 512, in call
    return self._run_internal_graph(inputs, training=training, mask=mask)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/functional.py", line 669, in _run_internal_graph
    outputs = node.layer(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1150, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/layers/convolutional/base_conv.py", line 321, in call
    return self.activation(outputs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/activations.py", line 321, in relu
    return backend.relu(

  File "/home/lnapon/repos/ancient_ai/lrnkeras/mnist_conv.py", line 25, in <module>
    model.fit(train_images, trains_labels, epochs=5, batch_size=1)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1768, in fit
    tmp_logs = self.train_function(iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1364, in train_function
    return step_function(self, iterator)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1348, in step_function
    outputs = model.distribute_strategy.run(run_step, args=(data,))

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1329, in run_step
    outputs = model.train_step(data)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 1106, in train_step
    y_pred = self(x, training=True)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/training.py", line 588, in __call__
    return super().__call__(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1150, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/sequential.py", line 404, in call
    return super().call(inputs, training=training, mask=mask)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/functional.py", line 512, in call
    return self._run_internal_graph(inputs, training=training, mask=mask)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/functional.py", line 669, in _run_internal_graph
    outputs = node.layer(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/engine/base_layer.py", line 1150, in __call__
    outputs = call_fn(inputs, *args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
    return fn(*args, **kwargs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/layers/convolutional/base_conv.py", line 321, in call
    return self.activation(outputs)

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/activations.py", line 321, in relu
    return backend.relu(

  File "/home/lnapon/miniconda3/envs/tf-nightly/lib/python3.10/site-packages/keras/src/backend.py", line 5397, in relu
    x = tf.nn.relu(x)

No algorithm worked!  Error messages:
  Profiling failure on CUDNN engine eng11{}: UNKNOWN: CUDNN_STATUS_ALLOC_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5333): 'status'
  Profiling failure on CUDNN engine eng0{}: UNKNOWN: CUDNN_STATUS_ALLOC_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5333): 'status'
	 [[{{node sequential/conv2d/Relu}}]] [Op:__inference_train_function_1058]

``` @O-CLIPE,
I tried to execute the mentioned code by importing the keras from tensorflow and it was executed without any issues on tensorflow v2.12. Kindly find the [gist](https://colab.research.google.com/gist/tilakrayal/faa9b889f64a401df67da39f8bb716b4/untitled1178.ipynb), [GPU-gist](https://colab.research.google.com/gist/tilakrayal/6fa70e52a2f49c52d3397f1b3da2fcba/untitled1179.ipynb) and below changes.
```

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.datasets import mnist
from tensorflow.keras.utils import to_categorical
from tensorflow.keras import layers, models
```
Thank you! @tilakrayal based on the error message, what could this error be about? I did get it is not a global tf issue, it's probably hardware specific. Also, with CPU only the code runs perfectly. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60713">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60713">No</a>
 It is hardware specific. Tensorflow drowns my GPU with memory usage and fails itself. To solve this I just limited the GPU memory usage, to lower than what I can expend (I overflows the limit a bit — in my case a bit means 24% of GPU memory).

2048MiB of memory if just too little for most applications of machine learning.
 Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60713">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60713">No</a>
 How to limit GPU memory usage: https://stackoverflow.com/a/60069601/20940588

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

Yes

### Source

source

### Tensorflow Version

tf 2.12

### Custom Code

Yes

### OS Platform and Distribution

Linux Ubuntu 22.04.2 LTS and Red Hat Enterprise Linux 9.2

### Mobile device

_No response_

### Python version

3.10

### Bazel version

bazel 5.3.0

### GCC/Compiler version

11.3.1

### CUDA/cuDNN version

11.8

### GPU model and memory

Nvidia Quadro K620 - 2048MiB of memory

### Current Behaviour?

*This bug happens both in Linux Ubuntu 22.04.2 LTS and RHEL 9. I tested it on a live-usb Ubuntu installation, pip installing it with GPU support after getting tired of trying to solve it on rhel.

It is somewhat related to the convolutional (Conv2d) layer, because fitting a model on a pip install with the use of only dense layers works fine. With a source install this error happens independently of model time.

Using CLI or multi-user.target, leaving all the GPU resources to tf does not solve the issue.

Reducing the Conv2d filters does not solve the problem, nor the bach_size

The code should just have no errors. I can't exactly tell or understand the error output, so that's about all I can tell.

### Standalone code to reproduce the issue

```shell
from keras.datasets import mnist
from keras.utils import to_categorical
from keras import layers, models

(train_images, train_labels), (test_images, test_labels) = mnist.load_data()

model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.Flatten())
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(10, activation='softmax'))

model.compile(optimizer="rmsprop", loss="categorical_crossentropy", metrics=['accuracy'])

train_images = train_images.astype('float32') / 255
test_images = test_images.astype('float32') / 255

trains_labels = to_categorical(train_labels)
test_labels = to_categorical(test_labels)

model.fit(train_images, trains_labels, epochs=5, batch_size=64)
test_loss, test_acc = model.evaluate(test_images, test_labels)
print(f'{test_acc=}')
```


### Relevant log output

```shell
(tf) [lnapon@rhel ancient_ai]$ python3 lrnkeras/mnist_conv.py 
2023-05-25 19:21:32.077966: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-05-25 19:21:33.793139: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-25 19:21:33.809742: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-25 19:21:33.809975: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-25 19:21:33.811051: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-25 19:21:33.811233: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-25 19:21:33.811384: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-25 19:21:34.315816: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-25 19:21:34.316045: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-25 19:21:34.316220: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-25 19:21:34.316380: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1294 MB memory:  -> device: 0, name: Quadro K620, pci bus id: 0000:01:00.0, compute capability: 5.0
Epoch 1/5
2023-05-25 19:21:35.567154: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:424] Loaded cuDNN version 8600
2023-05-25 19:21:35.886005: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:219] failed to create cublas handle: the resource allocation failed
2023-05-25 19:21:35.886043: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:222] Failure to initialize cublas may be due to OOM (cublas needs some free memory when you initialize it, and your deep-learning framework may have preallocated more than its fair share), or may be because this binary was not built with support for the GPU in your machine.
2023-05-25 19:21:35.886349: W tensorflow/core/kernels/conv_ops_gpu.cc:143] None of the algorithms provided by cuDNN frontend heuristics worked; trying fallback algorithms.  Conv: batch: 1
in_depths: 1
out_depths: 32
in: 28
in: 28
data_format: 1
filter: 3
filter: 3
filter: 1
dilation: 1
dilation: 1
stride: 1
stride: 1
padding: 0
padding: 0
dtype: DT_FLOAT
group_count: 1
device_identifier: "sm_5.0 with 2099118080B RAM, 3 cores, 1124000KHz clock, 900000KHz mem clock, 2097152B L2$"
fusion {
  activation_mode: kRelu
  conv_scale: 1
}
version: 3

2023-05-25 19:21:35.893161: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:219] failed to create cublas handle: the resource allocation failed
2023-05-25 19:21:35.893178: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:222] Failure to initialize cublas may be due to OOM (cublas needs some free memory when you initialize it, and your deep-learning framework may have preallocated more than its fair share), or may be because this binary was not built with support for the GPU in your machine.
2023-05-25 19:21:35.893443: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at conv_ops_fused_impl.h:625 : NOT_FOUND: No algorithm worked!  Error messages:
  Profiling failure on CUDNN engine eng11{}: UNKNOWN: CUDNN_STATUS_ALLOC_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(4639): 'status'
  Profiling failure on CUDNN engine eng0{}: UNKNOWN: CUDNN_STATUS_ALLOC_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(4639): 'status'
2023-05-25 19:21:35.893485: I tensorflow/core/common_runtime/executor.cc:1197] [/job:localhost/replica:0/task:0/device:GPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): NOT_FOUND: No algorithm worked!  Error messages:
  Profiling failure on CUDNN engine eng11{}: UNKNOWN: CUDNN_STATUS_ALLOC_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(4639): 'status'
  Profiling failure on CUDNN engine eng0{}: UNKNOWN: CUDNN_STATUS_ALLOC_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(4639): 'status'
	 [[{{node sequential/conv2d/Relu}}]]
Traceback (most recent call last):
  File "/home/lnapon/repos/ancient_ai/lrnkeras/mnist_conv.py", line 28, in <module>
    model.fit(train_images, trains_labels, epochs=5, batch_size=1)
  File "/home/lnapon/miniconda3/envs/tf/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/lnapon/miniconda3/envs/tf/lib/python3.10/site-packages/tensorflow/python/eager/execute.py", line 52, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.NotFoundError: Graph execution error:

Detected at node 'sequential/conv2d/Relu' defined at (most recent call last):
    File "/home/lnapon/repos/ancient_ai/lrnkeras/mnist_conv.py", line 28, in <module>
      model.fit(train_images, trains_labels, epochs=5, batch_size=1)
    File "/home/lnapon/miniconda3/envs/tf/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/home/lnapon/miniconda3/envs/tf/lib/python3.10/site-packages/keras/engine/training.py", line 1685, in fit
      tmp_logs = self.train_function(iterator)
    File "/home/lnapon/miniconda3/envs/tf/lib/python3.10/site-packages/keras/engine/training.py", line 1284, in train_function
      return step_function(self, iterator)
    File "/home/lnapon/miniconda3/envs/tf/lib/python3.10/site-packages/keras/engine/training.py", line 1268, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/home/lnapon/miniconda3/envs/tf/lib/python3.10/site-packages/keras/engine/training.py", line 1249, in run_step
      outputs = model.train_step(data)
    File "/home/lnapon/miniconda3/envs/tf/lib/python3.10/site-packages/keras/engine/training.py", line 1050, in train_step
      y_pred = self(x, training=True)
    File "/home/lnapon/miniconda3/envs/tf/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/home/lnapon/miniconda3/envs/tf/lib/python3.10/site-packages/keras/engine/training.py", line 558, in __call__
      return super().__call__(*args, **kwargs)
    File "/home/lnapon/miniconda3/envs/tf/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/home/lnapon/miniconda3/envs/tf/lib/python3.10/site-packages/keras/engine/base_layer.py", line 1145, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/home/lnapon/miniconda3/envs/tf/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 96, in error_handler
      return fn(*args, **kwargs)
    File "/home/lnapon/miniconda3/envs/tf/lib/python3.10/site-packages/keras/engine/sequential.py", line 412, in call
      return super().call(inputs, training=training, mask=mask)
    File "/home/lnapon/miniconda3/envs/tf/lib/python3.10/site-packages/keras/engine/functional.py", line 512, in call
      return self._run_internal_graph(inputs, training=training, mask=mask)
    File "/home/lnapon/miniconda3/envs/tf/lib/python3.10/site-packages/keras/engine/functional.py", line 669, in _run_internal_graph
      outputs = node.layer(*args, **kwargs)
    File "/home/lnapon/miniconda3/envs/tf/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/home/lnapon/miniconda3/envs/tf/lib/python3.10/site-packages/keras/engine/base_layer.py", line 1145, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/home/lnapon/miniconda3/envs/tf/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 96, in error_handler
      return fn(*args, **kwargs)
    File "/home/lnapon/miniconda3/envs/tf/lib/python3.10/site-packages/keras/layers/convolutional/base_conv.py", line 321, in call
      return self.activation(outputs)
    File "/home/lnapon/miniconda3/envs/tf/lib/python3.10/site-packages/keras/activations.py", line 317, in relu
      return backend.relu(
    File "/home/lnapon/miniconda3/envs/tf/lib/python3.10/site-packages/keras/backend.py", line 5396, in relu
      x = tf.nn.relu(x)
Node: 'sequential/conv2d/Relu'
No algorithm worked!  Error messages:
  Profiling failure on CUDNN engine eng11{}: UNKNOWN: CUDNN_STATUS_ALLOC_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(4639): 'status'
  Profiling failure on CUDNN engine eng0{}: UNKNOWN: CUDNN_STATUS_ALLOC_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(4639): 'status'
	 [[{{node sequential/conv2d/Relu}}]] [Op:__inference_train_function_1058]
```
</details>