It is unclear how a user should configure LD_LIBRARY_PATH after pip install tensorflow[and-cuda]
@jackhart We had a long discussion about it and figured it out: https://github.com/tensorflow/tensorflow/issues/63362

It's a path issue. Supposedly, there's a pull request where this gets fixed either in 2.16.2 or 2.17. @jackhart it seems practically impossible for someone owning a PC with CUDA-enabled GPU to perform deep learning experiments with TensorFlow version 2.16.1 and utilize his GPU locally without manually performing some extra steps not included (until today) in the official TensorFlow documentation of the standard installation procedure of TensorFlow for Linux users with GPUs at least as a temporal fix! That's why I submitted the pull request in good faith and for the shake of all users as TensorFlow is "_An Open Source Machine Learning Framework for Everyone_".

Hope that the next patch version of TensorFlow will fix the bug as soon as possible!

https://github.com/tensorflow/docs/pull/2299 I am having an incredibly frustrating experience. I followed the exact steps outlined here (https://www.tensorflow.org/install/pip), and despite my best efforts and hours of searching I keep getting "Cannot dlopen some GPU libraries" .. "Skipping registering GPU devices...".

I keep reading something about the LD_LIBRARY_PATH however I cannot find an obvious fix especially as I am using a python venv! I miss the days when tf-gpu worked on windows. This is pushing me to pytorch because I am exhausted trying different commands and path configurations to no avail. Why is it so difficult to use my GPU with Tensorflow? > I am having an incredibly frustrating experience. I followed the exact steps outlined here (https://www.tensorflow.org/install/pip), and despite my best efforts and hours of searching I keep getting "Cannot dlopen some GPU libraries" .. "Skipping registering GPU devices...".
> 
> I keep reading something about the LD_LIBRARY_PATH however I cannot find an obvious fix especially as I am using a python venv! I miss the days when tf-gpu worked on windows. This is pushing me to pytorch because I am exhausted trying different commands and path configurations to no avail. Why is it so difficult to use my GPU with Tensorflow?

@Edfrost321 a revised document addressing the issue of TensorFlow installation for Linux users with GPUs is pending review. Here is the link: https://github.com/sgkouzias/docs/blob/patch-1/site%2Fen%2Finstall%2Fpip.md 

I hope it helps! Hi **@jackhart** ,
Sorry for the delay, discussion is going on belongs to this issue already [#63362](https://github.com/tensorflow/tensorflow/issues/63362) , [#2299](https://github.com/tensorflow/docs/pull/2299). Once we get update, let you know.

Thank you! @sgkouzias 

That's a huge help, thank you! Thank you! Closing. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/65842">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/65842">No</a>

### Issue type

Build/Install

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

binary

### TensorFlow version

v2.16.1-0-g5bc9d26649c 2.16.1

### Custom code

No

### OS platform and distribution

Debian GNU/Linux 12 (bookworm)

### Mobile device

_No response_

### Python version

3.9.19

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

See below

### GPU model and memory

n/a

### Current behavior?

## Overview

It is unclear how a user should manage the CUDA dependencies installed with `pip install tensorflow[and-cuda]`.

LD_LIBRARY_PATH is used to load CUDA libs.  However, the install/install docs do not tell the user how to add the installed CUDA libs to the LD_LIBRARY_PATH.  I had to add every `venv/python*/site-packages/nvidia/<name>/lib` the  LD_LIBRARY_PATH, since the linker does not recursively search a directory.

## Dependencies 
* Container CUDA RuntimeVersion: 12.3
* CUDA Diver Version (what is linked by nvidia-smi - a nuance of containerized NVIDIA): 12.2
* Misc other NVIDIA dependencies installed with [and-cuda]: https://github.com/tensorflow/tensorflow/blob/v2.16.1/tensorflow/tools/pip_package/setup.py#L174

## Final Note

I might be missing something obvious.  But my impression is that the process required to use the pip installed CUDA dependencies is a bit involved and not documented.  I would appreciate any insight your teams' expertise can provide here!  Thank you! 


### Standalone code to reproduce the issue

```shell
======================== Dockerfile ========================
FROM python:3.9

ENV NVARCH=x86_64
ENV NVIDIA_VISIBLE_DEVICES=all
ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility
ENV LD_LIBRARY_PATH /usr/local/nvidia/lib:/usr/local/nvidia/lib64:${LD_LIBRARY_PATH}
ENV NVIDIA_REQUIRE_CUDA "cuda>=11.6, arch<9,driver>=450"

WORKDIR /app
ENV PATH /app/venv/bin:${PATH}
RUN python3 -m venv /app/venv && \
    pip install --upgrade pip setuptools wheel

RUN pip install tensorflow[and-cuda]

COPY entrypoint.sh /app/entrypoint.sh
ENTRYPOINT ["/app/entrypoint.sh"]

======================== entrypoint.sh ========================
#!/bin/bash
set -e
set -o xtrace

# Uncomment for solution.
# The LD_LIBRARY_PATH cannot have nested directories, so I link each pip installed nvidia lib.  
#export LD_LIBRARY_PATH=$(find /app/venv/lib -type d -path "*/python*/site-packages/nvidia/*" -prune -exec printf ':%s/lib' {} +):${LD_LIBRARY_PATH}

LD_DEBUG=libs python -c "import tensorflow as tf; print(f'Devices List: {tf.config.list_physical_devices()}')"


======================== command line from host ========================
docker build -t tensorflow-test .
docker run --gpus all tensorflow-test:latest
```


### Relevant log output

```shell
# ======================== logs when LD_LIBRARY_PATH isn't properly set ========================
+ python -c 'import tensorflow as tf; print(f'\''Devices List: {tf.config.list_physical_devices()}'\'')'
2024-04-16 13:49:18.866268: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-16 13:49:20.778286: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-04-16 13:49:22.322541: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2251] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
Devices List: [PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU')]

# ======================== logs when LD_LIBRARY_PATH is properly set ========================
+ LD_LIBRARY_PATH=:/app/venv/lib/python3.9/site-packages/nvidia/cudnn/lib:/app/venv/lib/python3.9/site-packages/nvidia/cuda_cupti/lib:/app/venv/lib/python3.9/site-packages/nvidia/cufft/lib:/app/venv/lib/python3.9/site-packages/nvidia/nccl/lib:/app/venv/lib/python3.9/site-packages/nvidia/__pycache__/lib:/app/venv/lib/python3.9/site-packages/nvidia/cublas/lib:/app/venv/lib/python3.9/site-packages/nvidia/nvjitlink/lib:/app/venv/lib/python3.9/site-packages/nvidia/cuda_runtime/lib:/app/venv/lib/python3.9/site-packages/nvidia/cuda_nvrtc/lib:/app/venv/lib/python3.9/site-packages/nvidia/cusolver/lib:/app/venv/lib/python3.9/site-packages/nvidia/cuda_nvcc/lib:/app/venv/lib/python3.9/site-packages/nvidia/curand/lib:/app/venv/lib/python3.9/site-packages/nvidia/cusparse/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:
+ python -c 'import tensorflow as tf; print(f'\''Devices List: {tf.config.list_physical_devices()}'\'')'
2024-04-16 13:50:41.598098: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-16 13:50:43.546095: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Devices List: [PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'), PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:2', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:3', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:4', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:5', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:6', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:7', device_type='GPU')]
```
