Get CUDA dependencies from PyPI packages
Hi @nluehr This PR is in draft, any update on this? Please. Thank you! @nluehr Nice, thanks! Internally we're talking about what we can do to bring this convenience to users. Using the new wheels somehow looks like a big improvement over [TensorFlow's miniconda-based instructions](https://www.tensorflow.org/install/pip#step-by-step_instructions).

I have some questions about the way the CUDA wheels will interact with TF's wheels and CUDA library discovery.

1. If the Bazel changes in this PR were to land without the wheel's explicit dependencies on those versioned CUDA packages, would the wheel be able to find the CUDA libraries as long as they are installed somewhere on the system or in the same venv? 
2. What happens if there are differently versioned CUDA wheels installed on the system already?
3. If the Bazel changes in this PR were to land without the wheel's explicit dependencies on those versioned CUDA packages, could the TensorFlow documentation simply say "install this list of CUDA packages in your system or venv" and it would always work?
4. Just to confirm, there's no equivalent, easy packages with the development headers for compiling TF, right? I'm assuming we can't use anything similar for building TF; just for the runtime.

If we can, I'm leaning towards an approach where we change the installation instructions to suggest a list of wheels in a Python env --- instead of putting CUDA dependencies directly in setup.py. If we add those dependencies, I can imagine some users may be surprised by the new behavior:  

  - Users who only want CPU support would always end up installing the large CUDA packages
  - Users who may have CUDA installed already would install it again, possibly confusing their system
  - Users who have another CUDA Python package installed somewhere would install them again, and it's not clear which would be used
  - If we add an extra-dependencies modifier to the wheel, like `tensorflow[cuda]`, then installation instructions, tutorials and requirements.txt files all over the ecosystem would get messy (i.e. simply depending on "tensorflow" is no longer the default behavior; requirements.txt files may start depending on tensorflow[cuda] which could then introduce the same problems as the previous bullets). Hi @angerson Thanks for taking a look. To answer your questions:

1. If the Bazel changes in this PR were to land without the wheel's dependencies, would the wheel be able to find CUDA libraries installed somewhere else on the system or in the same venv?
    For system packages, yes. The rpath mechanism prepends to the regular ld search paths (e.g., LD_LIBRARY_PATH, ldconfig paths, etc.).
    venv packages would also work (provided we're talking about nvidia's lib wheels), provided everything is in the same venv. The rpath mechanism assumes only that the cuda lib pip packages are installed adjacent to the tensorflow package. (Things would not work in a corner case such as installing the cuda libs among the the system packages while installing tensorflow in a "system-site-packages" venv.)
    Also, third party cuda pacakges (e.g., from conda) are likely packaged differently, so they would likely not be found unless explicitly added to the ld path.

2. What happens if there are differently versioned CUDA wheels installed on the system already?
    For the most part, Nvidia's wheels install into site-packages/nvidia/libname/lib/. I think other cuda wheels probably don't match this structure and would be ignored.

3. If the Bazel changes in this PR were to land without the wheel's explicit dependencies on those versioned CUDA packages, could the TensorFlow documentation simply say "install this list of CUDA packages in your system or venv" and it would always work?
    The package names differ depending on origin. I think today the docs specify the actual deb/rpm packages to install, the pip packages would need to be listed similarly. (otherwise someone might publish a wheel with a similar enough name to cause confusion). But in principle, it shouldn't matter which supported option the user chooses (official debs or official wheels). It should just work.

4. Just to confirm, there's no equivalent, easy packages with the development headers for compiling TF, right? I'm assuming we can't use anything similar for building TF; just for the runtime.
    Correct. These wheels are intended for runtime deployment only. Note however, that the headers are published separately at http://gitlab.com/nvidia/headers. You should be able to combine those with the wheel libs to get pretty close to what you have in mind. @angerson,

I've thought a bit about your proposal to include the rpaths but not the explicit dependencies. It makes sense that TensorFlow should not depend by default on the nvidia wheels (since, as you note, many use cases don't need them).

I think the chance of confusing the system is fairly small. I doubt there will be collisions with package locations such as `site-packages/nvidia/cublas/lib` and it's also unlikely the system loader would be configured to search such places when loading libs for non-tensorflow programs.

As far is it not being clear which library will be used, the rpath linker entries will place the wheel locations at the highest priority for the loader. So for tensorflow, the nvidia wheels will take precedence over both other system or wheel-packaged libs (as those all must depend on ldconfig changes or LD_LIBRARY_PATH).

I think an extra-dependencies modifier would be very helpful because users no longer need to concern themselves with the exact list of lib packages needed to enable cuda support (and docs wouldn't go out of date every time TensorFlow updates its cuda version). I don't think excluding an extra-deps option would avoid the downsides you mention. If tutorials, installation instructions, and requirements.txt files would be tempted to include the cuda extra deps modifier, are they going to be less likely to include the explicit list of (hard to remember) individual cuda wheels? If the include the latter, their docs are sure to go out of date as soon as TensorFlow updates its cuda version.

Perhaps it would help users if we give the extra-deps modifier a more verbose name, something like `tensorflow[with-cuda-wheel-libs]` so it's clear that this modifier isn't enabling core cuda support, but rather is a feature to easily fetching the needed libs. Hi @nluehr Can you please check @Artem-B's comments and keep us posted ? Thank you! Thanks guys. There are some minor lint issues that I am gonna try and just take care of myself later this afternoon to get it merged without any hassle.  It's taking a little longer than I expected due to internal/external differences. Update: I'm still having trouble with this. It's trickier than I thought so it's going slowly. > The rpath mechanism assumes only that the cuda lib pip packages are installed adjacent to the tensorflow package.

Hi, there is a way to load cuda lib pip packages in a different path, which even works in the current TF version:

```py
# TF 2.12 installed
import ctypes
from importlib import import_module,
from pathlib import Path

def dlopen_library(module: str, filename: str):
    try:
        m = import_module(module)
    except ModuleNotFoundError:
        pass
    else:
        libs = sorted(Path(m.__file__).parent.glob(filename))
        # hope that there is only one version installed...
        if len(libs):
            ctypes.CDLL(str(libs[0].absolute()))

# dlopen pip cuda library before tensorflow
if platform.system() == "Linux":
    dlopen_library("nvidia.cuda_runtime.lib", "libcudart.so*")
    dlopen_library("nvidia.cublas.lib", "libcublasLt.so*")
    dlopen_library("nvidia.cublas.lib", "libcublas.so*")
    dlopen_library("nvidia.cufft.lib", "libcufft.so*")
    dlopen_library("nvidia.curand.lib", "libcurand.so*")
    dlopen_library("nvidia.cusolver.lib", "libcusolver.so*")
    dlopen_library("nvidia.cusparse.lib", "libcusparse.so*")
    dlopen_library("nvidia.cudnn.lib", "libcudnn.so*")

import tensorflow as tf
```

The critical point is to use `importlib.import_module` to get the absolute path of the installed CUDA libraries. I'd like to know if this will be helpful. Hi @nluehr Can you please resolve conflicts? Thank you! Hi @nluehr Can you please resolve conflicts? Thank you! Rebased to resolve merge conflicts.
@angerson PTAL.
 @nluehr @trevor-m Thanks a lot for helping un-stick this. I know the changes were small but I had no idea what to do about them. As part of the merge, I had to make some adjustments to get everything working in our CI and export system -- hopefully nothing I did was incorrect. The merge should come through pretty soon and we should be able to get some data along with it. Success! The latest versions of tf-nightly can now be amended with `[and-cuda]`. @MichaelHudgins FYI. The next step to make this land in TF 2.14 is just to ensure that the list of extras is kept up-to-date if the TF team upgrades CUDA versions.

```
docker run -it --rm --gpus=all nvidia/cuda:11.8.0-base-ubuntu22.04 bash
> apt update
> apt install python3-pip
> pip install tf-nightly[and-cuda]
> python3
>> import tensorflow as tf
>> tf.debugging.set_log_device_placement(True)
>> a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])
>> b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])
>> c = tf.matmul(a, b)
>> print(c)
```
@trevor-m @nluehr Thanks a lot for contributing this and helping me fix the import issues! FYI, the logs for my test above complained about "Unable to register cuDNN (and cuBLAS and cuFFT) factory", as seen here: https://gist.github.com/angerson/8058444763245554436b0604ea42aaa8. That's not related to this change, right? > Success! The latest versions of tf-nightly can now be amended with `[and-cuda]`. @MichaelHudgins FYI. The next step to make this land in TF 2.14 is just to ensure that the list of extras is kept up-to-date if the TF team upgrades CUDA versions.
> 
> ```
> docker run -it --rm --gpus=all nvidia/cuda:11.8.0-base-ubuntu22.04 bash
> > apt update
> > apt install python3-pip
> > pip install tf-nightly[and-cuda]
> > python3
> >> import tensorflow as tf
> >> tf.debugging.set_log_device_placement(True)
> >> a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])
> >> b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])
> >> c = tf.matmul(a, b)
> >> print(c)
> ```
> 
> @trevor-m @nluehr Thanks a lot for contributing this and helping me fix the import issues! FYI, the logs for my test above complained about "Unable to register cuDNN (and cuBLAS and cuFFT) factory", as seen here: https://gist.github.com/angerson/8058444763245554436b0604ea42aaa8. That's not related to this change, right?

Thanks a bunch @angerson! No, those are unrelated warnings due to duplicate registrations. They already existed and can be safely ignored. @fchollet What's the context for this getting re-opened? @angerson never mind. There it is! https://www.tensorflow.org/install/pip
This PR is a POC demonstrating building TensorFlow so that it depends on NVIDIA's wheel-based library releases for the required CUDA libraries and runtime. Users will still need to install the CUDA driver separately, but all other NVIDIA library dependencies should be fetched automatically by pip.

For example, after building the wheel with the usual OSS build scripts (based on cuda 11.8), the resulting wheel could be installed in a fairly minimalist ubuntu 20.04 environment as follows:

```
docker run --rm -it --gpus all -v /path/to/tensorflow/pkg:/tf/pkg nvidia/cuda:12.0.1-base-ubuntu20.04
apt update && apt install -y --no-install-recommends curl python3 python3-pip
python3 -mpip install --upgrade pip
pip3 install /tf/pkg/tensorflow-*.whl
```

Notice that the CUDA driver's forward compatibility allows users to use the latest (here CUDA 12) base container/driver, while pip pulls in the necessary CUDA 11.8-based libraries as needed.

Currently, NVIDIA wheels are released only for linux platforms and the x86_64 architecture.