Tesla v100 Tensorflow CUDA Support
Hi @anand-shubham ,

Please find the tested configurations of CUDA and cuDNN [here](https://www.tensorflow.org/install/source#gpu).


tensorflow-2.4.0 | 3.6-3.8 | GCC 7.3.1 | Bazel 3.1.0 | 8.0 | 11.0
-- | -- | -- | -- | -- | --

For TF2.4v you should use CUDA 11.0 and cuDNN 8.0 and GCC versions should be 7.3.1.

Please upgrade GCC to 7.3.1 and downgrade CUDA and cuDNN to above mentioned configurations because we can't assure backward compatibility of latest CUDA/cuDNN for older TF versions.

Please do above changes and let us know if still having problem.

Thanks!

 Hi @SuryanarayanaY,

While installing the Nvidia driver we have an option to select the CUDA TOOLKIT version. Does that affect the CUDA, cuDNN we install later?
As I have already tried with the  For TF2.4v you should use CUDA 11.0 and cuDNN 8.0.

The GPU installed is Tesla V100 Volta Architecture. 

Current RHEL Version :Red Hat Enterprise Linux Server release 7.9 (Maipo) 

Current GCC Verison: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44) . Hi @anand-shubham ,

As per official documentation it recommends to install CUDA and cuDNN toolkits using Conda explicitly. As you are using TF2.4 version which is quite older versions which we are not supporting now.Also official documentation keeping only the instructions for latest TF release which you can find [here](https://www.tensorflow.org/install/pip#step-by-step_instructions). Please note that the attached instructions are applicable to latest TF version which is Tf2.13 now.

I am attaching the instructions which I have used for Tf2.11 .

```
conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/
mkdir -p $CONDA_PREFIX/etc/conda/activate.d
echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/' > $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
```
You can try replacing with `cudatoolkit=11.0 cudnn=8.0` and follow same instructions for path setting mentioned above.

First you need to install GPU driver and verify it using `nvidia-smi` command and then follow the instructions mentioned above for CUDA/cuDNN toolkit installation and path setting.

Could you also please check your GPU compatibility from Nvidia website may be from [here](https://developer.nvidia.com/cuda-gpus). 

But we recommend to use latest TF versions that can be better supported by us. Thanks!
 2023-07-24 18:29:10.026158: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:428] Loaded cuDNN version 8100
2023-07-24 18:29:40.514393: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:219] failed to create cublas handle: cublas error
2023-07-24 18:29:40.514447: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:221] Failure to initialize cublas may be due to OOM (cublas needs some free memory when you initialize it, and your deep-learning framework may have preallocated more than its fair share), or may be because this binary was not built with support for the GPU in your machine.
2023-07-24 18:29:40.514769: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at conv_ops_fused_impl.h:621 : NOT_FOUND: No algorithm worked!  Error messages:
  Profiling failure on CUDNN engine 1#TC: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 1: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 0#TC: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 0: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 2#TC: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 2: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 4#TC: UNKNOWN: CUDNN_STATUS_INTERNAL_ERROR
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 4: UNKNOWN: CUDNN_STATUS_INTERNAL_ERROR
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 6#TC: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 6: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 5#TC: UNKNOWN: CUDNN_STATUS_INTERNAL_ERROR
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 5: UNKNOWN: CUDNN_STATUS_INTERNAL_ERROR
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 7#TC: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 7: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
Traceback (most recent call last):
  File "/data01/PatternClassClassification/Model_2_PatternClass_imgSize150x150.py", line 366, in <module>
    history = stn_model.fit(
  File "/data01/Software_installations/Anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/data01/Software_installations/Anaconda3/envs/gpu_env/lib/python3.10/site-packages/tensorflow/python/eager/execute.py", line 52, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.NotFoundError: Graph execution error:

Detected at node 'model_1/conv2d_5/Relu' defined at (most recent call last):
    File "/data01/PatternClassClassification/Model_2.py", line 366, in <module>
      history = stn_model.fit(
    File "/data01/Software_installations/Anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/data01/Software_installations/Anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/engine/training.py", line 1650, in fit
      tmp_logs = self.train_function(iterator)
    File "/data01/Software_installations/Anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/engine/training.py", line 1249, in train_function
      return step_function(self, iterator)
    File "/data01/Software_installations/Anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/engine/training.py", line 1233, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/data01/Software_installations/Anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/engine/training.py", line 1222, in run_step
      outputs = model.train_step(data)
    File "/data01/Software_installations/Anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/engine/training.py", line 1023, in train_step
      y_pred = self(x, training=True)
    File "/data01/Software_installations/Anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/data01/Software_installations/Anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/engine/training.py", line 561, in __call__
      return super().__call__(*args, **kwargs)
    File "/data01/Software_installations/Anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/data01/Software_installations/Anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/engine/base_layer.py", line 1132, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/data01/Software_installations/Anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 96, in error_handler
      return fn(*args, **kwargs)
    File "/data01/Software_installations/Anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/engine/functional.py", line 511, in call
      return self._run_internal_graph(inputs, training=training, mask=mask)
    File "/data01/Software_installations/Anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/engine/functional.py", line 668, in _run_internal_graph
      outputs = node.layer(*args, **kwargs)
    File "/data01/Software_installations/Anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/data01/Software_installations/Anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/engine/base_layer.py", line 1132, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/data01/Software_installations/Anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 96, in error_handler
      return fn(*args, **kwargs)
    File "/data01/Software_installations/Anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/layers/convolutional/base_conv.py", line 314, in call
      return self.activation(outputs)
    File "/data01/Software_installations/Anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/activations.py", line 317, in relu
      return backend.relu(
    File "/data01/Software_installations/Anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/backend.py", line 5369, in relu
      x = tf.nn.relu(x)
Node: 'model_1/conv2d_5/Relu'
No algorithm worked!  Error messages:
  Profiling failure on CUDNN engine 1#TC: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 1: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 0#TC: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 0: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 2#TC: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 2: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 4#TC: UNKNOWN: CUDNN_STATUS_INTERNAL_ERROR
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 4: UNKNOWN: CUDNN_STATUS_INTERNAL_ERROR
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 6#TC: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 6: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 5#TC: UNKNOWN: CUDNN_STATUS_INTERNAL_ERROR
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 5: UNKNOWN: CUDNN_STATUS_INTERNAL_ERROR
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 7#TC: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
  Profiling failure on CUDNN engine 7: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(5150): 'status'
         [[{{node model_1/conv2d_5/Relu}}]] [Op:__inference_train_function_1992]
2023-07-24 18:29:40.809535: W tensorflow/core/kernels/data/generator_dataset_op.cc:108] Error occurred when finalizing GeneratorDataset iterator: FAILED_PRECONDITION: Python interpreter state is not initialized. The process may be terminated.
         [[{{node PyFunc}}]] > 2023-07-24 18:29:40.514447: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:221] Failure to initialize cublas may be due to OOM (cublas needs some free memory when you initialize it, and your deep-learning framework may have preallocated more than its fair share), or may be because this binary was not built with support for the GPU in your machine.

@anand-shubham , Could you please refer to the above error and comment. This seems not an issue with tensorflow. WARNING:tensorflow:`period` argument is deprecated. Please use `save_freq` to specify the frequency in number of batches seen.
Epoch 1/200
2023-07-26 16:31:45.274459: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:428] Loaded cuDNN version 8100
2023-07-26 16:33:46.507694: I tensorflow/tsl/platform/default/subprocess.cc:304] Start cannot spawn child process: No such file or directory
2023-07-26 16:33:46.508174: I tensorflow/tsl/platform/default/subprocess.cc:304] Start cannot spawn child process: No such file or directory
2023-07-26 16:33:46.508206: W tensorflow/compiler/xla/stream_executor/gpu/asm_compiler.cc:85] Couldn't get ptxas version string: INTERNAL: Couldn't invoke ptxas --version
2023-07-26 16:33:46.508519: I tensorflow/tsl/platform/default/subprocess.cc:304] Start cannot spawn child process: No such file or directory
2023-07-26 16:33:46.508591: W tensorflow/compiler/xla/stream_executor/gpu/redzone_allocator.cc:318] INTERNAL: Failed to launch ptxas
Relying on driver to perform ptx compilation.
Modify $PATH to customize ptxas location.
This message will be only logged once.
2023-07-26 16:38:08.228703: I tensorflow/compiler/xla/service/service.cc:173] XLA service 0x1e177620 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2023-07-26 16:38:08.228760: I tensorflow/compiler/xla/service/service.cc:181]   StreamExecutor device (0): GRID V100L-8Q, Compute Capability 7.0
2023-07-26 16:38:08.234554: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
2023-07-26 16:38:08.256595: W tensorflow/compiler/xla/service/gpu/nvptx_helper.cc:56] Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice. This may result in compilation or runtime failures, if the program we try to run uses routines from libdevice.
Searched for CUDA in the following directories:
  ./cuda_sdk_lib
  /usr/local/cuda-11.2
  /usr/local/cuda
  .
You can choose the search directory by setting xla_gpu_cuda_data_dir in HloModule's DebugOptions.  For most apps, setting the environment variable XLA_FLAGS=--xla_gpu_cuda_data_dir=/path/to/cuda will work.
2023-07-26 16:38:08.257645: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:326] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-07-26 16:38:08.257856: I tensorflow/compiler/jit/xla_compilation_cache.cc:477] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
2023-07-26 16:38:08.257985: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:446 : INTERNAL: libdevice not found at ./libdevice.10.bc
2023-07-26 16:38:08.280213: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:326] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-07-26 16:38:08.280541: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:446 : INTERNAL: libdevice not found at ./libdevice.10.bc
2023-07-26 16:38:09.127454: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:326] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-07-26 16:38:09.127784: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:446 : INTERNAL: libdevice not found at ./libdevice.10.bc
2023-07-26 16:38:09.149194: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:326] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-07-26 16:38:09.149491: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:446 : INTERNAL: libdevice not found at ./libdevice.10.bc
2023-07-26 16:39:35.064225: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:326] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-07-26 16:39:35.064572: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:446 : INTERNAL: libdevice not found at ./libdevice.10.bc
2023-07-26 16:39:35.085963: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:326] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-07-26 16:39:35.086272: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:446 : INTERNAL: libdevice not found at ./libdevice.10.bc
2023-07-26 16:39:47.420330: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:326] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-07-26 16:39:47.420685: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:446 : INTERNAL: libdevice not found at ./libdevice.10.bc
2023-07-26 16:39:47.442196: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:326] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-07-26 16:39:47.442501: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:446 : INTERNAL: libdevice not found at ./libdevice.10.bc
Traceback (most recent call last):
  File "/data01/PatternClassClassification/test_gpu1.py", line 71, in <module>
    history = model.fit(
  File "/data01/Software_installations/anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/data01/Software_installations/anaconda3/envs/gpu_env/lib/python3.10/site-packages/tensorflow/python/eager/execute.py", line 52, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InternalError: Graph execution error:

Detected at node 'StatefulPartitionedCall_6' defined at (most recent call last):
    File "/data01/PatternClassClassification/test_gpu1.py", line 71, in <module>
      history = model.fit(
    File "/data01/Software_installations/anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/data01/Software_installations/anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/engine/training.py", line 1650, in fit
      tmp_logs = self.train_function(iterator)
    File "/data01/Software_installations/anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/engine/training.py", line 1249, in train_function
      return step_function(self, iterator)
    File "/data01/Software_installations/anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/engine/training.py", line 1233, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/data01/Software_installations/anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/engine/training.py", line 1222, in run_step
      outputs = model.train_step(data)
    File "/data01/Software_installations/anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/engine/training.py", line 1027, in train_step
      self.optimizer.minimize(loss, self.trainable_variables, tape=tape)
    File "/data01/Software_installations/anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 527, in minimize
      self.apply_gradients(grads_and_vars)
    File "/data01/Software_installations/anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1140, in apply_gradients
      return super().apply_gradients(grads_and_vars, name=name)
    File "/data01/Software_installations/anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 634, in apply_gradients
      iteration = self._internal_apply_gradients(grads_and_vars)
    File "/data01/Software_installations/anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1166, in _internal_apply_gradients
      return tf.__internal__.distribute.interim.maybe_merge_call(
    File "/data01/Software_installations/anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1216, in _distributed_apply_gradients_fn
      distribution.extended.update(
    File "/data01/Software_installations/anaconda3/envs/gpu_env/lib/python3.10/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1211, in apply_grad_to_update_var
      return self._update_step_xla(grad, var, id(self._var_key(var)))
Node: 'StatefulPartitionedCall_6'
libdevice not found at ./libdevice.10.bc
         [[{{node StatefulPartitionedCall_6}}]] [Op:__inference_train_function_1141]
2023-07-26 16:39:47.722164: W tensorflow/core/kernels/data/generator_dataset_op.cc:108] Error occurred when finalizing GeneratorDataset iterator: FAILED_PRECONDITION: Python interpreter state is not initialized. The process may be terminated.
         [[{{node PyFunc}}]]



The code runs with the CPU version of Tensorflow

The model is compiled and I can see the Architecture as the output, but when the training is about to start the program crashes Hi @anand-shubham ,

Since you are using TF2.4v which is not supported anymore, can you please try with any of latest TF versions and let us know if issue still persists on latest versions ?

Also we officially supports Ubuntu instructions only which may work for other linux distros also. I can see from your inputs its not problem with Ubuntu right ? In that case it may not be possible to look into this issue unless this is a problem with latest TF versions also.

Please try with latest TF version and let us know the outcome. Thanks! hi @SuryanarayanaY ,

The above output is with TF2.11v Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61330">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61330">No</a>
 @SuryanarayanaY  after installing CUDA cuDNN by installing without conda the CNN has worked, but there is no speedup is that a Tensorflow issue?

The ETA for 1 epoch is much higher than what I get when running on CPU. Hi @anand-shubham ,

Could you please submit a code snippet along with logs to test it on Ubuntu?

Thanks!

 Also it seems now issue is quite different than earlier.If earlier issue resolved and current problem is different it would be better to create a new ticket if you are ok with it.

Thanks! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61330">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61330">No</a>

### Issue type

Support

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

source

### TensorFlow version

2.4

### Custom code

Yes

### OS platform and distribution

RHEL 7.9

### Mobile device

_No response_

### Python version

3.6

### Bazel version

_No response_

### GCC/compiler version

4.3

### CUDA/cuDNN version

11/8, 10.1/7.6

### GPU model and memory

Tesla V100 2GB Vram

### Current behavior?

Attempting to fetch value instead of handling error Internal: failed to get device attribute 13 for device 0: CUDA_ERROR_UNKNOWN: unknown error.

NVIDIA-SMI give the following output:| NVIDIA-SMI 450.51.05 Driver Version: 450.51.05 CUDA Version: 11.0

nvcc-V the following output:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Fri_Feb__8_19:08:17_PST_2019
Cuda compilation tools, release 10.1, V10.1.105


### Standalone code to reproduce the issue

```shell
Doesnt happen with Windows or Ubuntu systems.
```


### Relevant log output

_No response_