Failed to load in-memory CUBIN: CUDA_ERROR_NO_BINARY_FOR_GPU: no kernel image is available for execution on the device 	 [[node Generador/conv2d_4/Tanh] [Op:__inference_predict_function_1081]
@ivolis,
There are changes in the paths with the latest versions and I have changed the target to **tensorflow/core/kernels/mlir_generated:gpu_binary_ops_test.cc** and tested the bazel build which was success. 

There have been changes in paths compared to your earlier tested targets and bugs fixed in recent versions. Also you are trying to install tensorflow v2.4 with CUDA 12.1 which is not compatible. Please have a look at the official tested build configurations and follow the instructions. 
https://www.tensorflow.org/install/source#gpu
Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60530">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60530">No</a>
 Hi,
I am running into the same problem with NVIDIA A100 80GB. But same code and Conda environment works in Tesla V100 16GB. Both having same cuda version and nvidia driver.
What was the solution?

More details:
It works in:                        Graphic card: Tesla V100 16GB,    CUDA Version: 11.4, Driver version: 470.161.03
It has the CUBIN error in: Graphic card: NVIDIA A100 80GB, CUDA Version: 11.4, Driver Version: 470.161.03
Conda Environment (same for both): Python 3.9.16, tensorflow 2.4.1

Thanks
Hi everyone,

I'm currently using tensorflow in order to use a GAN Network I coded myself (custom code). I am quite new when it comes to Linux and TF.

# Relevant information
- Version of TensorFlow: 2.4.1 (installed using `pip install tensorflow-gpu`)
- OS:  Zorin OS 16.2 (Ubuntu 20.04) 
- Python version: 3.9.16 
- CUDA version: 12.1
- GPU : NVIDIA RTX A2000 12GB 
- Nvidia driver version: 530.30.02
- CUDNN version: 8.9.0

# Problem

My model is created  and stored on the GPU memory correctly (with the following **warnings**)

```
2023-05-08 09:37:30.383063: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2023-05-08 09:37:31.619864: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2023-05-08 09:37:31.630697: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2023-05-08 09:37:31.692075: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-05-08 09:37:31.692323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA RTX A2000 12GB computeCapability: 8.6
coreClock: 1.2GHz coreCount: 26 deviceMemorySize: 11.75GiB deviceMemoryBandwidth: 268.26GiB/s
2023-05-08 09:37:31.692349: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2023-05-08 09:37:31.756689: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10
2023-05-08 09:37:31.756793: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.10
2023-05-08 09:37:31.792431: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2023-05-08 09:37:31.801029: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2023-05-08 09:37:31.866009: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2023-05-08 09:37:31.874931: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10
2023-05-08 09:37:31.988965: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.7
2023-05-08 09:37:31.989159: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-05-08 09:37:31.989419: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-05-08 09:37:31.989544: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2023-05-08 09:37:31.990621: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-05-08 09:37:31.993043: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2023-05-08 09:37:31.993197: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-05-08 09:37:31.993380: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: NVIDIA RTX A2000 12GB computeCapability: 8.6
coreClock: 1.2GHz coreCount: 26 deviceMemorySize: 11.75GiB deviceMemoryBandwidth: 268.26GiB/s
2023-05-08 09:37:31.993406: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2023-05-08 09:37:31.993436: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10
2023-05-08 09:37:31.993451: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.10
2023-05-08 09:37:31.993467: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2023-05-08 09:37:31.993484: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2023-05-08 09:37:31.993502: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2023-05-08 09:37:31.993518: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10
2023-05-08 09:37:31.993535: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.7
2023-05-08 09:37:31.993600: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-05-08 09:37:31.993763: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-05-08 09:37:31.993881: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2023-05-08 09:37:31.994703: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2023-05-08 09:39:47.663704: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2023-05-08 09:39:47.663723: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2023-05-08 09:39:47.663727: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2023-05-08 09:39:47.664624: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-05-08 09:39:47.664707: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-05-08 09:39:47.664753: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-05-08 09:39:47.664800: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10694 MB memory) -> physical GPU (device: 0, name: NVIDIA RTX A2000 12GB, pci bus id: 0000:01:00.0, compute capability: 8.6)
```

I think these are common warnings. Besides, I printed the summary after this and it showed correctly (see extra information at the end of the issue).

After that, when make some **prediction** to make some sort of visual control over the generator images, the output looks like this:

```
2023-05-08 09:39:47.890873: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2023-05-08 09:39:47.918809: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2112000000 Hz
2023-05-08 09:39:48.043219: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10
2023-05-08 09:40:39.040740: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.7
2023-05-08 09:42:07.469816: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at cwise_op_gpu_base.cc:89 : Internal: Failed to load in-memory CUBIN: CUDA_ERROR_NO_BINARY_FOR_GPU: no kernel image is available for execution on the device
Traceback (most recent call last):
  File "/home/calculin/Desktop/DCGAN_MELUS-main/GAN_train.py", line 110, in <module>
    aux_generated_images = generator.predict(aux_noise, verbose = 0)   #Create the images from the GAN.    
  File "/home/calculin/miniconda3/envs/tensorflow/lib/python3.9/site-packages/tensorflow/python/keras/engine/training.py", line 1629, in predict
    tmp_batch_outputs = self.predict_function(iterator)
  File "/home/calculin/miniconda3/envs/tensorflow/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 828, in __call__
    result = self._call(*args, **kwds)
  File "/home/calculin/miniconda3/envs/tensorflow/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 894, in _call
    return self._concrete_stateful_fn._call_flat(
  File "/home/calculin/miniconda3/envs/tensorflow/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 1918, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "/home/calculin/miniconda3/envs/tensorflow/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 555, in call
    outputs = execute.execute(
  File "/home/calculin/miniconda3/envs/tensorflow/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InternalError:  Failed to load in-memory CUBIN: CUDA_ERROR_NO_BINARY_FOR_GPU: no kernel image is available for execution on the device
	 [[node Generador/conv2d_4/Tanh (defined at /Desktop/DCGAN_MELUS-main/GAN_train.py:110) ]] [Op:__inference_predict_function_1081]

Function call stack:
predict_function
```

**And then the kernel just dies. I am running it on terminal just by doing `python3 mycode.py`**

## [Context: this is the code runned in order to get the error]
```
aux_noise = np.random.normal(0, 1, size=(5, noise_dim))
aux_generated_images = generator.predict(aux_noise, verbose = 0)   #Create the images from the GAN.
```



# Some extra information from terminal:

```
(tensorflow) calculin@pcGAN:~$ dpkg -l | grep cudnn
ii  cudnn-local-repo-ubuntu2004-8.9.0.131                  1.0-1                                       amd64        cudnn-local repository configuration files
ii  libcudnn8                                              8.9.0.131-1+cuda12.1                        amd64        cuDNN runtime libraries
ii  libcudnn8-dev                                          8.9.0.131-1+cuda12.1                        amd64        cuDNN development libraries and headers
ii  libcudnn8-samples                                      8.9.0.131-1+cuda12.1                        amd64        cuDNN samples
```

```
(tensorflow) calculin@pcGAN:~$ dpkg -l | grep cuda
ii  cuda                                                   12.1.1-1                                    amd64        CUDA meta-package
ii  cuda-12-1                                              12.1.1-1                                    amd64        CUDA 12.1 meta-package
ii  cuda-cccl-12-1                                         12.1.109-1                                  amd64        CUDA CCCL
ii  cuda-command-line-tools-12-1                           12.1.1-1                                    amd64        CUDA command-line tools
ii  cuda-compiler-12-1                                     12.1.1-1                                    amd64        CUDA compiler
ii  cuda-cudart-12-1                                       12.1.105-1                                  amd64        CUDA Runtime native Libraries
ii  cuda-cudart-dev-12-1                                   12.1.105-1                                  amd64        CUDA Runtime native dev links, headers
ii  cuda-cuobjdump-12-1                                    12.1.111-1                                  amd64        CUDA cuobjdump
ii  cuda-cupti-12-1                                        12.1.105-1                                  amd64        CUDA profiling tools runtime libs.
ii  cuda-cupti-dev-12-1                                    12.1.105-1                                  amd64        CUDA profiling tools interface.
ii  cuda-cuxxfilt-12-1                                     12.1.105-1                                  amd64        CUDA cuxxfilt
ii  cuda-demo-suite-12-1                                   12.1.105-1                                  amd64        Demo suite for CUDA
ii  cuda-documentation-12-1                                12.1.105-1                                  amd64        CUDA documentation
ii  cuda-driver-dev-12-1                                   12.1.105-1                                  amd64        CUDA Driver native dev stub library
ii  cuda-drivers                                           530.30.02-1                                 amd64        CUDA Driver meta-package, branch-agnostic
ii  cuda-drivers-530                                       530.30.02-1                                 amd64        CUDA Driver meta-package, branch-specific
ii  cuda-gdb-12-1                                          12.1.105-1                                  amd64        CUDA-GDB
ii  cuda-libraries-12-1                                    12.1.1-1                                    amd64        CUDA Libraries 12.1 meta-package
ii  cuda-libraries-dev-12-1                                12.1.1-1                                    amd64        CUDA Libraries 12.1 development meta-package
ii  cuda-nsight-12-1                                       12.1.105-1                                  amd64        CUDA nsight
ii  cuda-nsight-compute-12-1                               12.1.1-1                                    amd64        NVIDIA Nsight Compute
ii  cuda-nsight-systems-12-1                               12.1.1-1                                    amd64        NVIDIA Nsight Systems
ii  cuda-nvcc-12-1                                         12.1.105-1                                  amd64        CUDA nvcc
ii  cuda-nvdisasm-12-1                                     12.1.105-1                                  amd64        CUDA disassembler
ii  cuda-nvml-dev-12-1                                     12.1.105-1                                  amd64        NVML native dev links, headers
ii  cuda-nvprof-12-1                                       12.1.105-1                                  amd64        CUDA Profiler tools
ii  cuda-nvprune-12-1                                      12.1.105-1                                  amd64        CUDA nvprune
ii  cuda-nvrtc-12-1                                        12.1.105-1                                  amd64        NVRTC native runtime libraries
ii  cuda-nvrtc-dev-12-1                                    12.1.105-1                                  amd64        NVRTC native dev links, headers
ii  cuda-nvtx-12-1                                         12.1.105-1                                  amd64        NVIDIA Tools Extension
ii  cuda-nvvp-12-1                                         12.1.105-1                                  amd64        CUDA Profiler tools
ii  cuda-opencl-12-1                                       12.1.105-1                                  amd64        CUDA OpenCL native Libraries
ii  cuda-opencl-dev-12-1                                   12.1.105-1                                  amd64        CUDA OpenCL native dev links, headers
ii  cuda-profiler-api-12-1                                 12.1.105-1                                  amd64        CUDA Profiler API
ii  cuda-repo-ubuntu2004-12-1-local                        12.1.1-530.30.02-1                          amd64        cuda repository configuration files
ii  cuda-runtime-12-1                                      12.1.1-1                                    amd64        CUDA Runtime 12.1 meta-package
ii  cuda-sanitizer-12-1                                    12.1.105-1                                  amd64        CUDA Sanitizer
ii  cuda-toolkit-12-1                                      12.1.1-1                                    amd64        CUDA Toolkit 12.1 meta-package
ii  cuda-toolkit-12-1-config-common                        12.1.105-1                                  all          Common config package for CUDA Toolkit 12.1.
ii  cuda-toolkit-12-config-common                          12.1.105-1                                  all          Common config package for CUDA Toolkit 12.
ii  cuda-toolkit-config-common                             12.1.105-1                                  all          Common config package for CUDA Toolkit.
ii  cuda-tools-12-1                                        12.1.1-1                                    amd64        CUDA Tools meta-package
ii  cuda-visual-tools-12-1                                 12.1.1-1                                    amd64        CUDA visual tools
ii  libcudart10.1:amd64                                    10.1.243-3                                  amd64        NVIDIA CUDA Runtime Library
ii  libcudnn8                                              8.9.0.131-1+cuda12.1                        amd64        cuDNN runtime libraries
ii  libcudnn8-dev                                          8.9.0.131-1+cuda12.1                        amd64        cuDNN development libraries and headers
ii  libcudnn8-samples                                      8.9.0.131-1+cuda12.1                        amd64        cuDNN samples
ii  nvidia-cuda-dev                                        10.1.243-3                                  amd64        NVIDIA CUDA development files
ii  nvidia-cuda-doc                                        10.1.243-3                                  all          NVIDIA CUDA and OpenCL documentation
ii  nvidia-cuda-gdb                                        10.1.243-3                                  amd64        NVIDIA CUDA Debugger (GDB)
ii  nvidia-cuda-toolkit                                    10.1.243-3                                  amd64        NVIDIA CUDA development toolkit
```

```
(tensorflow) calculin@inti-013308:~$ ls /usr/local | grep cuda
cuda
cuda-12
cuda-12.1
```

```
(tensorflow) calculin@inti-013308:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Mon_Apr__3_17:16:06_PDT_2023
Cuda compilation tools, release 12.1, V12.1.105
Build cuda_12.1.r12.1/compiler.32688072_0
```


```
Model: "Discriminador"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d (Conv2D)              (None, 128, 128, 64)      640       
_________________________________________________________________
batch_normalization (BatchNo (None, 128, 128, 64)      256       
_________________________________________________________________
leaky_re_lu (LeakyReLU)      (None, 128, 128, 64)      0         
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 64, 64, 128)       73856     
_________________________________________________________________
batch_normalization_1 (Batch (None, 64, 64, 128)       512       
_________________________________________________________________
leaky_re_lu_1 (LeakyReLU)    (None, 64, 64, 128)       0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 32, 32, 128)       147584    
_________________________________________________________________
batch_normalization_2 (Batch (None, 32, 32, 128)       512       
_________________________________________________________________
leaky_re_lu_2 (LeakyReLU)    (None, 32, 32, 128)       0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 16, 16, 256)       295168    
_________________________________________________________________
batch_normalization_3 (Batch (None, 16, 16, 256)       1024      
_________________________________________________________________
leaky_re_lu_3 (LeakyReLU)    (None, 16, 16, 256)       0         
_________________________________________________________________
flatten (Flatten)            (None, 65536)             0         
_________________________________________________________________
dropout (Dropout)            (None, 65536)             0         
_________________________________________________________________
dense (Dense)                (None, 1)                 65537     
=================================================================
Total params: 585,089
Trainable params: 583,937
Non-trainable params: 1,152
_________________________________________________________________
Model: "Generador"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_1 (Dense)              (None, 65536)             6619136   
_________________________________________________________________
batch_normalization_4 (Batch (None, 65536)             262144    
_________________________________________________________________
leaky_re_lu_4 (LeakyReLU)    (None, 65536)             0         
_________________________________________________________________
reshape (Reshape)            (None, 16, 16, 256)       0         
_________________________________________________________________
conv2d_transpose (Conv2DTran (None, 32, 32, 128)       524416    
_________________________________________________________________
batch_normalization_5 (Batch (None, 32, 32, 128)       512       
_________________________________________________________________
leaky_re_lu_5 (LeakyReLU)    (None, 32, 32, 128)       0         
_________________________________________________________________
conv2d_transpose_1 (Conv2DTr (None, 64, 64, 128)       262272    
_________________________________________________________________
batch_normalization_6 (Batch (None, 64, 64, 128)       512       
_________________________________________________________________
leaky_re_lu_6 (LeakyReLU)    (None, 64, 64, 128)       0         
_________________________________________________________________
conv2d_transpose_2 (Conv2DTr (None, 128, 128, 64)      131136    
_________________________________________________________________
batch_normalization_7 (Batch (None, 128, 128, 64)      256       
_________________________________________________________________
leaky_re_lu_7 (LeakyReLU)    (None, 128, 128, 64)      0         
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 128, 128, 1)       577       
=================================================================
Total params: 7,800,961
Trainable params: 7,669,249
Non-trainable params: 131,712
_________________________________________________________________
Model: "GAN_completa"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 100)]             0         
_________________________________________________________________
Generador (Sequential)       (None, 128, 128, 1)       7800961   
_________________________________________________________________
Discriminador (Sequential)   (None, 1)                 585089    
=================================================================
Total params: 8,386,050
Trainable params: 7,669,249
Non-trainable params: 716,801
```