CUDA_ERROR_NOT_PERMITTED: operation not permitted when fitting model on GPU
@pawelremiszewski,
I was facing a different error while executing the mentioned code. Kindly find the gist of it [here](https://colab.research.google.com/gist/tilakrayal/66ba5681e65198b272d4c880ed71f1cd/untitled1215.ipynb) and provide the complete dependencies. 

Also the above error occurs when there’s a tensor / cuda object getting created or transferred inside a recorded graph. All CUDA objects need to be initialized on the GPU prior to recording a graph. Could you please try to  shut off Cuda graphs in your `config.yaml` using **cuda_graphs = False.** 

[tf.keras.preprocessing.image.ImageDataGenerator](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator) is not recommended for new code. Prefer loading images with [tf.keras.utils.image_dataset_from_directory](https://www.tensorflow.org/api_docs/python/tf/keras/utils/image_dataset_from_directory) and transforming the output [tf.data.Dataset](https://www.tensorflow.org/api_docs/python/tf/data/Dataset) with preprocessing layers. For more information, see the tutorials for [loading images](https://www.tensorflow.org/tutorials/load_data/images) and [augmenting images](https://www.tensorflow.org/tutorials/images/data_augmentation), as well as the [preprocessing layer guide](https://www.tensorflow.org/guide/keras/preprocessing_layers).

Thank you! @tilakrayal 
Could You please tell me, what more You need in code, to check the correct error?
It is quite obvious, that You got "NameError: name 'Y_train' is not defined", if the variable is not specified, as it is numpy array created from data that I have.
What is more, as I see, in untitled1215.ipynb You are using "2.12.0" version of tensorflow, in error is 2.10.

Question, "config.yaml" - where can I find this file? I can't find it anywhere.

I cannot use [tf.keras.utils.image_dataset_from_directory], as I have my own dataset in one folder - images and txt files with labels fot them. 
I tried to put it all into tf.dataset using tf.data.Dataset.from_tensor_slices, but it caused higher memory usage on GPU, and also it caused input_shape problem in first Conv2D layer, so I dropped this idea


After all, I found the issue that caused the error.
Before, I used this code to make it possible for GPU to free memory constantly, to avoid lack of memory:
"os.environ['TF_GPU_ALLOCATOR'] = 'cuda_malloc_async'
print(os.getenv('TF_GPU_ALLOCATOR'))"
I removed it, and error from the title is now showing anymore.

Now my model starts to fit, but after 1st Epoch, it throws me this:
Epoch 1/5
219/219 [==============================] - ETA: 0s - loss: 1.6949 - accuracy: 0.3793   Traceback (most recent call last):
  Cell In[4], line 3
    model.fit(
  File ~\anaconda3\envs\pawel_engineer\lib\site-packages\keras\utils\traceback_utils.py:70 in error_handler
    raise e.with_traceback(filtered_tb) from None
  File ~\anaconda3\envs\pawel_engineer\lib\site-packages\tensorflow\python\eager\execute.py:54 in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
ResourceExhaustedError: Graph execution error:
Detected at node 'sequential/max_pooling2d/MaxPool' defined at (most recent call last): ......"
and at the end:
bfc_allocator.cc:1101] Sum Total of in-use chunks: 2.07GiB
2023-06-27 18:53:15.410883: I tensorflow/core/common_runtime/bfc_allocator.cc:1103] total_region_allocated_bytes_: 2236245504 memory_limit_: 2236245607 available bytes: 103 curr_region_allocation_bytes_: 2147483648
2023-06-27 18:53:15.410902: I tensorflow/core/common_runtime/bfc_allocator.cc:1109] Stats: 
Limit:                      2236245607
InUse:                      2218346496
MaxInUse:                   2221506304
NumAllocs:                       23816
MaxAllocSize:               1374683136
Reserved:                            0
PeakReserved:                        0
LargestFreeBlock:                    0

2023-06-27 18:53:15.410912: W tensorflow/core/common_runtime/bfc_allocator.cc:491] ****************************************************************************************************
2023-06-27 18:53:15.410945: W tensorflow/core/framework/op_kernel.cc:1780] OP_REQUIRES failed at pooling_ops_common.cc:226 : RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[32,63,63,128] and type half on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc @pawelremiszewski,
Apologies for the delay. The error facing was "NameError: name 'Y_train' is not defined" where the Y_train was not mentioned in the given code. 

Also OOM stands for "out of memory". Your GPU is running out of memory, so it can't allocate memory for this tensor. There are a few things you can do:

- Decrease the number of filters in your Dense, Conv2D layers
- Use a smaller batch_size (or increase steps_per_epoch and validation_steps)
- Use grayscale images (you can use [tf.image.rgb_to_grayscale](https://www.tensorflow.org/api_docs/python/tf/image/rgb_to_grayscale))
- Reduce the number of layers
- Use **MaxPooling2D** layers after convolutional layers
- Reduce the size of your images (you can use [tf.image.resize](https://www.tensorflow.org/api_docs/python/tf/image/resize) for that)
- Use smaller float precision for your input, namely **np.float32**

Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61076">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61076">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

No

### Source

source

### Tensorflow Version

v2.10.0-rc3-6-g359c3cdfc5f 2.10.0

### Custom Code

Yes

### OS Platform and Distribution

Windows 10

### Mobile device

_No response_

### Python version

3.9.16

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

cuDNN version 8.1 CUDA version 11.2

### GPU model and memory

name: NVIDIA GeForce GTX 1650 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5, memory: 4 GB

### Current Behaviour?

A bug happened!
I have a problem with fitting my model.
Additional informations:
This model is about to recognize images. There are 10 classes to predict, co I'm using loss=categorical_crossentropy and last activation Layer is 'softmax'
My input data shapes are:
print(X_train.shape) -> (3495, 128, 128, 3)
print(Y_train.shape) -> (3495, 10)
print(X_val.shape) -> (1498, 128, 128, 3)
print(Y_val.shape) -> (1498, 10)
So it is quite small for test.

My GPU is visible - I tested it with:
model.add(Activation('sigmoid')) and loss='binary_crossentropy' first and it is running.
I have set memory allocation and growth as in code. Everything seemed fine.

Now when I changed it to:
model.add(Activation('softmax')) and loss='categorical_crossentropy ' - also used 'to_categorical()' for classes array - I'm getting such error:
Failed setting context: CUDA_ERROR_NOT_PERMITTED: operation not permitted.

Looking at my GPU usage, it doesn't look like it is a memory problem - in peak it is not using even 30% of it.
Did anyone ever get such error? I run out of ideas.



### Standalone code to reproduce the issue

```shell
gpus = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(gpus[0], True)
tf.config.set_visible_devices(gpus[:1], device_type='GPU')
log_dev_conf = tf.config.LogicalDeviceConfiguration(
    memory_limit=3*512
)
tf.config.set_logical_device_configuration(
    gpus[0],
    [log_dev_conf]) 

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation, Flatten
from tensorflow.keras.layers import Conv2D, MaxPooling2D
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.utils import to_categorical

from tensorflow.keras.mixed_precision import set_global_policy
set_global_policy('mixed_float16')

tf.keras.backend.clear_session()

Y_train = to_categorical(Y_train, num_classes=10, dtype=int)
Y_val = to_categorical(Y_val, num_classes=10, dtype=int)

batch_size = 2
data_generator = ImageDataGenerator(rescale=1.0/255.0)
data_generator = data_generator.flow(X_train, Y_train, batch_size=batch_size)

data_generator_val = ImageDataGenerator(rescale=1.0/255.0)
data_generator_val = data_generator_val.flow(X_val, Y_val, batch_size=batch_size)


model = Sequential()

model.add(Conv2D(128, (3, 3), input_shape=X_train.shape[1:]))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(128, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Flatten())

model.add(Dense(64))
model.add(Activation('relu'))

model.add(Dense(10, dtype='float32'))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy',
              optimizer = 'Adam',
              metrics=['accuracy'])


# Train the model using the fit method
trained_model = model.fit(
    data_generator,
    validation_data=data_generator_val,
    steps_per_epoch=batch_size,
    epochs=5
)
```


### Relevant log output

```shell
Epoch 1/5

2023-06-25 19:13:24.452341: I tensorflow/stream_executor/cuda/cuda_dnn.cc:384] Loaded cuDNN version 8100
2023-06-25 19:13:28.511859: F tensorflow/stream_executor/cuda/cuda_driver.cc:147] Failed setting context: CUDA_ERROR_NOT_PERMITTED: operation not permitted


Fatal Python error: Aborted


Main thread:
Thread 0x00000eb8 (most recent call first):
  File "C:\Users\pablitoremiszewski\anaconda3\envs\pawel_engineer\lib\site-packages\tensorflow\python\eager\execute.py", line 54 in quick_execute
  File "C:\Users\pablitoremiszewski\anaconda3\envs\pawel_engineer\lib\site-packages\tensorflow\python\eager\function.py", line 499 in call
  File "C:\Users\pablitoremiszewski\anaconda3\envs\pawel_engineer\lib\site-packages\tensorflow\python\eager\function.py", line 1862 in _call_flat
  File "C:\Users\pablitoremiszewski\anaconda3\envs\pawel_engineer\lib\site-packages\tensorflow\python\eager\function.py", line 2496 in __call__
  File "C:\Users\pablitoremiszewski\anaconda3\envs\pawel_engineer\lib\site-packages\tensorflow\python\eager\def_function.py", line 980 in _call
  File "C:\Users\pablitoremiszewski\anaconda3\envs\pawel_engineer\lib\site-packages\tensorflow\python\eager\def_function.py", line 915 in __call__
  File "C:\Users\pablitoremiszewski\anaconda3\envs\pawel_engineer\lib\site-packages\tensorflow\python\util\traceback_utils.py", line 150 in error_handler
  File "C:\Users\pablitoremiszewski\anaconda3\envs\pawel_engineer\lib\site-packages\keras\engine\training.py", line 1564 in fit
  File "C:\Users\pablitoremiszewski\anaconda3\envs\pawel_engineer\lib\site-packages\keras\utils\traceback_utils.py", line 65 in error_handler
  File "C:\Users\pablitoremiszewski\AppData\Local\Temp\ipykernel_6540\2599062935.py", line 3 in <module>


Restarting kernel...
```
</details>