W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH
@monajalal,
Tensorflow v2.8.0 is compatible with python 3.7-3.10, GCC compiler - 7.3.1, Bazel - 4.2.1, cuDNN 8.1 and  CUDA 11.2. Could you please try to install tensorflow v2.8 with the tested build configurations.
https://www.tensorflow.org/install/source#gpu

Thank you!



 @tilakrayal 
I have no idea what may have changed but I ran the code again today and it's working. Any thoughts?

```
(samurai) mona@ard-gpu-01:~/samurai$ python train_samurai.py --config configs/samurai/samurai.txt --datadir data/duck/ --basedir . --expname duck_test --gpu 0
Namespace(config=None, basedir='.', expname='duck_test', batch_size=1024, learning_rate=0.0001, epochs=150, steps_per_epoch=2000, gpu='0', tpu=None, debug=False, profile=False, perturb=1.0, raw_noise_std=0.0, coarse_samples=64, linear_disparity_sampling=False, fine_samples=128, fourier_frequency=10, direction_fourier_frequency=4, random_encoding_offsets=True, fine_net_width=128, fine_net_depth=8, coarse_net_width=128, coarse_net_depth=6, appearance_latent_dim=32, diffuse_latent_dim=24, fix_diffuse=True, camera_distribution='sphere', use_fully_random_cameras=False, random_cameras_per_view=4, min_softmax_scaler=1.0, max_softmax_scaler=10.0, camera_weight_update_lr=0.3, camera_weight_update_momentum=0.75, bounding_size=0.5, resolution_factor=4, advanced_loss_done=80000, network_gradient_norm_clipping=0.1, camera_gradient_norm_clipping=-1, not_learn_r=False, not_learn_t=False, not_learn_f=False, edge_align_step=200, num_edge_align_steps=50, pretrained_camera_poses_folder=None, start_f_optimization=90000, start_fourier_anneal=0, finish_fourier_anneal=50000, slow_scheduler_decay=100000, brdf_schedule_decay=40000, lambda_smoothness=0.01, smoothness_bound_dividier=200, coarse_distortion_lambda=0.001, fine_distortion_lambda=0, normal_direction_lambda=0.005, mlp_normal_direction_lambda=0.0003, disable_posterior_scaling=False, disable_mask_uncertainty=True, lambda_brdf_decoder_smoothness=0.1, lambda_brdf_decoder_sparsity=0.01, camera_lr=0.003, camera_lr_decay=70, camera_regularization=0.1, aim_center_regularization=10.0, camera_rotation='lookat', learn_camera_offsets=True, basecolor_metallic=True, skip_decomposition=False, compose_on_white=True, rotating_object=False, single_env=False, brdf_preintegration_path='data/neural_pil/BRDFLut.hdr', illumination_network_path='data/neural_pil/illumination-network', datadir='data/duck/', max_resolution_dimension=400, test_holdout=16, dataset='samurai', load_gt_poses=False, canonical_pose=0, log_step=100, weights_epoch=5, validation_epoch=5, testset_epoch=150, video_epoch=50, lrate_decay=300, render_only=False)
2023-06-21 08:35:27.209395: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-06-21 08:35:27.246961: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib:/home/mona/MVTec/HALCON-23.05-Progress//lib/x64-linux:/usr/local/cuda-11.7/lib64:/home/mona/onnx-tensorrt/build:
2023-06-21 08:35:27.246980: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
Utilizing 0 GPUs for training.
2023-06-21 08:35:28.412840: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
(70, 3)
Model: "sequential_12"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 MappingNetwork/Layer_0 (Den  (None, 128)              16512     
 se)                                                             
                                                                 
 MappingNetwork/Layer_1 (Den  (None, 128)              16512     
 se)                                                             
                                                                 
 MappingNetwork/Final (Dense  (None, 768)              99072     
 )                                                               
                                                                 
 reshape_1 (Reshape)         (None, 2, 3, 128)         0         
                                                                 
=================================================================
Total params: 132,096
Trainable params: 132,096
Non-trainable params: 0
_________________________________________________________________
Model: "sequential_13"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 ConditionalNetwork/Dense1 (  (None, 32)               192       
 Dense)                                                          
                                                                 
 ConditionalNetwork/DenseFin  (None, 256)              8448      
 al (Dense)                                                      
                                                                 
 reshape_2 (Reshape)         (None, 2, 128)            0         
                                                                 
=================================================================
Total params: 8,640
Trainable params: 8,640
Non-trainable params: 0
_________________________________________________________________
Found ckpts []
Starting training in epoch 0 at step 0
Start Training...
/home/mona/anaconda3/envs/samurai/lib/python3.9/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/interpolate_bilinear/gather-bottom_right/GatherV2_grad/Reshape_1:0", shape=(1024,), dtype=int32), values=Tensor("gradients/interpolate_bilinear/gather-bottom_right/GatherV2_grad/Reshape:0", shape=(1024, 1), dtype=float32), dense_shape=Tensor("gradients/interpolate_bilinear/gather-bottom_right/GatherV2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/home/mona/anaconda3/envs/samurai/lib/python3.9/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/interpolate_bilinear/gather-bottom_left/GatherV2_grad/Reshape_1:0", shape=(1024,), dtype=int32), values=Tensor("gradients/interpolate_bilinear/gather-bottom_left/GatherV2_grad/Reshape:0", shape=(1024, 1), dtype=float32), dense_shape=Tensor("gradients/interpolate_bilinear/gather-bottom_left/GatherV2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/home/mona/anaconda3/envs/samurai/lib/python3.9/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/interpolate_bilinear/gather-top_right/GatherV2_grad/Reshape_1:0", shape=(1024,), dtype=int32), values=Tensor("gradients/interpolate_bilinear/gather-top_right/GatherV2_grad/Reshape:0", shape=(1024, 1), dtype=float32), dense_shape=Tensor("gradients/interpolate_bilinear/gather-top_right/GatherV2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/home/mona/anaconda3/envs/samurai/lib/python3.9/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/interpolate_bilinear/gather-top_left/GatherV2_grad/Reshape_1:0", shape=(1024,), dtype=int32), values=Tensor("gradients/interpolate_bilinear/gather-top_left/GatherV2_grad/Reshape:0", shape=(1024, 1), dtype=float32), dense_shape=Tensor("gradients/interpolate_bilinear/gather-top_left/GatherV2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
```
![Screenshot from 2023-06-21 08-48-56](https://github.com/tensorflow/tensorflow/assets/1892917/77813399-a426-49df-b0da-2c3f24be4c00)
 nevermind still have the error despite training going on
2023-06-21 08:35:27.246961: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib:/home/mona/MVTec/HALCON-23.05-Progress//lib/x64-linux:/usr/local/cuda-11.7/lib64:/home/mona/onnx-tensorrt/build:

Also I am using Python 3.9 not 3.7 Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60854">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60854">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

Yes

### Source

source

### Tensorflow Version

2.8.0

### Custom Code

No

### OS Platform and Distribution

Ubuntu 22.04

### Mobile device

_No response_

### Python version

Python 3.9.16 | packaged by conda-forge | (main, Feb  1 2023, 21:39:03) 

### Bazel version

bazel 5.3.2

### GCC/Compiler version

gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0

### CUDA/cuDNN version

11.2/8 in conda env

### GPU model and memory

laptop 3080 RTX

### Current Behaviour?

A bug happened!

### Standalone code to reproduce the issue

```shell
I have tensorflow 2 installed and also from the code below I see cudnn 8 is found. 


(samurai) mona@ard-gpu-01:~/samurai$ cat cudnn_test.py 
import tensorflow as tf

sys_details = tf.sysconfig.get_build_info()
cuda_version = sys_details["cuda_version"]
print(cuda_version)

cudnn_version = sys_details["cudnn_version"]
print(cudnn_version)


cuda_compute_capabilities = sys_details["cuda_compute_capabilities"]
print(cuda_compute_capabilities)
(samurai) mona@ard-gpu-01:~/samurai$ python cudnn_test.py 
11.2
8
['sm_35', 'sm_50', 'sm_60', 'sm_70', 'sm_75', 'compute_80']
```

However, when I run the following command, I get an error that cudnn 8 is not found.

```
(samurai) mona@ard-gpu-01:~/samurai$ python train_samurai.py --config configs/samurai/samurai.txt --datadir data/duck/ --basedir . --expname duck_test --gpu 0
Namespace(config=None, basedir='.', expname='duck_test', batch_size=1024, learning_rate=0.0001, epochs=150, steps_per_epoch=2000, gpu='0', tpu=None, debug=False, profile=False, perturb=1.0, raw_noise_std=0.0, coarse_samples=64, linear_disparity_sampling=False, fine_samples=128, fourier_frequency=10, direction_fourier_frequency=4, random_encoding_offsets=True, fine_net_width=128, fine_net_depth=8, coarse_net_width=128, coarse_net_depth=6, appearance_latent_dim=32, diffuse_latent_dim=24, fix_diffuse=True, camera_distribution='sphere', use_fully_random_cameras=False, random_cameras_per_view=4, min_softmax_scaler=1.0, max_softmax_scaler=10.0, camera_weight_update_lr=0.3, camera_weight_update_momentum=0.75, bounding_size=0.5, resolution_factor=4, advanced_loss_done=80000, network_gradient_norm_clipping=0.1, camera_gradient_norm_clipping=-1, not_learn_r=False, not_learn_t=False, not_learn_f=False, edge_align_step=200, num_edge_align_steps=50, pretrained_camera_poses_folder=None, start_f_optimization=90000, start_fourier_anneal=0, finish_fourier_anneal=50000, slow_scheduler_decay=100000, brdf_schedule_decay=40000, lambda_smoothness=0.01, smoothness_bound_dividier=200, coarse_distortion_lambda=0.001, fine_distortion_lambda=0, normal_direction_lambda=0.005, mlp_normal_direction_lambda=0.0003, disable_posterior_scaling=False, disable_mask_uncertainty=True, lambda_brdf_decoder_smoothness=0.1, lambda_brdf_decoder_sparsity=0.01, camera_lr=0.003, camera_lr_decay=70, camera_regularization=0.1, aim_center_regularization=10.0, camera_rotation='lookat', learn_camera_offsets=True, basecolor_metallic=True, skip_decomposition=False, compose_on_white=True, rotating_object=False, single_env=False, brdf_preintegration_path='data/neural_pil/BRDFLut.hdr', illumination_network_path='data/neural_pil/illumination-network', datadir='data/duck/', max_resolution_dimension=400, test_holdout=16, dataset='samurai', load_gt_poses=False, canonical_pose=0, log_step=100, weights_epoch=5, validation_epoch=5, testset_epoch=150, video_epoch=50, lrate_decay=300, render_only=False)
2023-06-13 15:35:10.002485: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-06-13 15:35:10.022702: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib:/home/mona/MVTec/HALCON-23.05-Progress//lib/x64-linux:/usr/local/cuda-11.7/lib64:/home/mona/onnx-tensorrt/build:
2023-06-13 15:35:10.022715: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
Utilizing 0 GPUs for training.
2023-06-13 15:35:11.092766: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
(70, 3)
Model: "sequential_12"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 MappingNetwork/Layer_0 (Den  (None, 128)              16512     
 se)                                                             
                                                                 
 MappingNetwork/Layer_1 (Den  (None, 128)              16512     
 se)                                                             
                                                                 
 MappingNetwork/Final (Dense  (None, 768)              99072     
 )                                                               
                                                                 
 reshape_1 (Reshape)         (None, 2, 3, 128)         0         
                                                                 
=================================================================
Total params: 132,096
Trainable params: 132,096
Non-trainable params: 0
_________________________________________________________________
Model: "sequential_13"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 ConditionalNetwork/Dense1 (  (None, 32)               192       
 Dense)                                                          
                                                                 
 ConditionalNetwork/DenseFin  (None, 256)              8448      
 al (Dense)                                                      
                                                                 
 reshape_2 (Reshape)         (None, 2, 128)            0         
                                                                 
=================================================================
Total params: 8,640
Trainable params: 8,640
Non-trainable params: 0
_________________________________________________________________
Found ckpts []
Starting training in epoch 0 at step 0
Start Training...
/home/mona/anaconda3/envs/samurai/lib/python3.9/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/interpolate_bilinear/gather-bottom_right/GatherV2_grad/Reshape_1:0", shape=(1024,), dtype=int32), values=Tensor("gradients/interpolate_bilinear/gather-bottom_right/GatherV2_grad/Reshape:0", shape=(1024, 1), dtype=float32), dense_shape=Tensor("gradients/interpolate_bilinear/gather-bottom_right/GatherV2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/home/mona/anaconda3/envs/samurai/lib/python3.9/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/interpolate_bilinear/gather-bottom_left/GatherV2_grad/Reshape_1:0", shape=(1024,), dtype=int32), values=Tensor("gradients/interpolate_bilinear/gather-bottom_left/GatherV2_grad/Reshape:0", shape=(1024, 1), dtype=float32), dense_shape=Tensor("gradients/interpolate_bilinear/gather-bottom_left/GatherV2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/home/mona/anaconda3/envs/samurai/lib/python3.9/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/interpolate_bilinear/gather-top_right/GatherV2_grad/Reshape_1:0", shape=(1024,), dtype=int32), values=Tensor("gradients/interpolate_bilinear/gather-top_right/GatherV2_grad/Reshape:0", shape=(1024, 1), dtype=float32), dense_shape=Tensor("gradients/interpolate_bilinear/gather-top_right/GatherV2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/home/mona/anaconda3/envs/samurai/lib/python3.9/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/interpolate_bilinear/gather-top_left/GatherV2_grad/Reshape_1:0", shape=(1024,), dtype=int32), values=Tensor("gradients/interpolate_bilinear/gather-top_left/GatherV2_grad/Reshape:0", shape=(1024, 1), dtype=float32), dense_shape=Tensor("gradients/interpolate_bilinear/gather-top_left/GatherV2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
  25/2000 [..............................] - ETA: 42:41 - loss: 1.8824 - loss_camera: 7.2076 - fine_loss: 1.8019   

```
```


### Relevant log output

```shell
(samurai) mona@ard-gpu-01:~/samurai$ lsb_release -a
LSB Version:	core-11.1.0ubuntu4-noarch:security-11.1.0ubuntu4-noarch
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.2 LTS
Release:	22.04
Codename:	jammy
(samurai) mona@ard-gpu-01:~/samurai$ uname -a
Linux ard-gpu-01 5.19.0-43-generic #44~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon May 22 13:39:36 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
```

```
(samurai) mona@ard-gpu-01:~/samurai$ nvidia-smi
Tue Jun 13 15:38:44 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3080 L...    On | 00000000:01:00.0 Off |                  N/A |
| N/A   49C    P8               17W /  90W|    102MiB / 16384MiB |     21%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      2549      G   /usr/lib/xorg/Xorg                           95MiB |
|    0   N/A  N/A      2983      G   ...libexec/gnome-remote-desktop-daemon        3MiB |
+---------------------------------------------------------------------------------------+
```
```
(samurai) mona@ard-gpu-01:~/samurai$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Jun__8_16:49:14_PDT_2022
Cuda compilation tools, release 11.7, V11.7.99
Build cuda_11.7.r11.7/compiler.31442593_0

```


The code is from this repo: https://github.com/google/samurai
```
</details>