Not able to build TensorFlow with GPU support
I too am failing to build TensorFlow for GPU (with or without the container).

The problem I see for you is that TensorRT 10 is not supported, the latest supported version is 8.6.1, see https://github.com/tensorflow/tensorflow/issues/66473. You have to install an older version and hopefully get past that error.

The problems I am getting are:
* When using GCC it gets all the way to near the end and fails to link saying some symbols cannot be found (they are symbols of template classes). This occurs even with `--noincompatible_do_not_split_linking_cmdline` as recommended by https://github.com/tensorflow/tensorflow/issues/35623.
* When using Clang it failed with `unused-command-line-argument` errors, but that was fixed by adding `--host_copt=-Wno-error=unused-command-line-argument`. After that it gets further but fails with messages about `__float128` not being a supported type on the target (the target being the GPU).
* In the container it cannot find nvml.h - that header does not exist in the container at all. > The problem I see for you is that TensorRT 10 is not supported, the latest supported version is 8.6.1,

Idk why the TensorFlow project is not updated frequently as PyTorch.

PyTorch even added support to the latest Ubuntu 24.04 Release and both Nvidia and TensorFlow are lagging behind.
### Issue type

Build/Install

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

2.16

### Custom code

Yes

### OS platform and distribution

Ubuntu LTS 22.04

### Mobile device

_No response_

### Python version

3.10.12

### Bazel version

6.5.0

### GCC/compiler version

Clang 17.0.6

### CUDA/cuDNN version

12.4/8.9.7

### GPU model and memory

RTX 4060 Laptop 8GB

### Current behavior?

I am builing TF r2.16 so that it supports CUDA and TensorRT. It is failing I don't know why. I used to use TF on windows bbut recently switched to linux to get GPU support.

`nvidia-smi`

```
Thu May 30 13:23:41 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14              Driver Version: 550.54.14      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4060 ...    Off |   00000000:01:00.0  On |                  N/A |
| N/A   43C    P8              4W /   80W |     342MiB /   8188MiB |      4%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      1098      G   /usr/lib/xorg/Xorg                            133MiB |
|    0   N/A  N/A      1536      G   budgie-wm                                      61MiB |
|    0   N/A  N/A      2150      G   ...irefox/3779/usr/lib/firefox/firefox          0MiB |
+-----------------------------------------------------------------------------------------+
```

All the code which I ran and all the error outputs are shown below.
Thanks

### Standalone code to reproduce the issue

```shell
sudo apt install python3-dev python3-pip
wget https://github.com/llvm/llvm-project/releases/download/llvmorg-17.0.2/clang+llvm-17.0.2-x86_64-linux-gnu-ubuntu-22.04.tar.xz
tar -xvf clang+llvm-17.0.2-x86_64-linux-gnu-ubuntu-22.04.tar.xz
cp -r clang+llvm-17.0.2-x86_64-linux-gnu-ubuntu-22.04/* /usr
git clone https://github.com/tensorflow/tensorflow.git
cd tensorflow
git checkout r2.16
./configure
bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package --jobs=20
```


### Relevant log output

```shell
./configure
"""
You have bazel 6.5.0 installed.
Please specify the location of python. [Default is /usr/bin/python3]: 


Found possible Python library paths:
  /usr/lib/python3.10/dist-packages
  /usr/lib/python3/dist-packages
  /usr/local/lib/python3.10/dist-packages
Please input the desired Python library path to use.  Default is [/usr/lib/python3.10/dist-packages]

Do you wish to build TensorFlow with ROCm support? [y/N]: n
No ROCm support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: y
CUDA support will be enabled for TensorFlow.

Do you wish to build TensorFlow with TensorRT support? [y/N]: y
TensorRT support will be enabled for TensorFlow.

Found CUDA 12.4 in:
    /usr/local/cuda-12.4/targets/x86_64-linux/lib
    /usr/local/cuda-12.4/targets/x86_64-linux/include
Found cuDNN 8 in:
    /usr/lib/x86_64-linux-gnu
    /usr/include
Found TensorRT 10.0.1 in:
    /usr/lib/x86_64-linux-gnu
    /usr/include/x86_64-linux-gnu


Please specify a list of comma-separated CUDA compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus. Each capability can be specified as "x.y" or "compute_xy" to include both virtual and binary GPU code, or as "sm_xy" to only include the binary code.
Please note that each additional compute capability significantly increases your build time and binary size, and that TensorFlow only supports compute capabilities >= 3.5 [Default is: 3.5,7.0]: 


Do you want to use clang as CUDA compiler? [Y/n]: y
Clang will be used as CUDA compiler.

Please specify clang path that to be used as host compiler. [Default is /usr/lib/llvm-17/bin/clang]: 


You have Clang 17.0.6 installed.

Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -Wno-sign-compare]: 


Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: n
Not configuring the WORKSPACE for Android builds.

Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See .bazelrc for more details.
	--config=mkl         	# Build with MKL support.
	--config=mkl_aarch64 	# Build with oneDNN and Compute Library for the Arm Architecture (ACL).
	--config=monolithic  	# Config for mostly static monolithic build.
	--config=numa        	# Build with NUMA support.
	--config=dynamic_kernels	# (Experimental) Build kernels into separate shared objects.
	--config=v1          	# Build with TensorFlow 1 API instead of TF 2 API.
Preconfigured Bazel build configs to DISABLE default on features:
	--config=nogcp       	# Disable GCP support.
	--config=nonccl      	# Disable NVIDIA NCCL support.
Configuration finished
"""

bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package --jobs=20
"""
WARNING: The following configs were expanded more than once: [tensorrt, cuda_clang, cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
INFO: Reading 'startup' options from /home/manveer/Downloads/tensorflow/.bazelrc: --windows_enable_symlinks
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=99
INFO: Reading rc options for 'build' from /home/manveer/Downloads/tensorflow/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /home/manveer/Downloads/tensorflow/.bazelrc:
  'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --features=-force_no_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility
INFO: Reading rc options for 'build' from /home/manveer/Downloads/tensorflow/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=/usr/bin/python3 --action_env PYTHON_LIB_PATH=/usr/lib/python3.10/dist-packages --python_path=/usr/bin/python3 --config=tensorrt --action_env TF_CUDA_PATHS=/usr/local/cuda-12.4,/usr --action_env CUDA_TOOLKIT_PATH=/usr/local/cuda-12.4 --action_env TF_CUDA_COMPUTE_CAPABILITIES=3.5,7.0 --action_env LD_LIBRARY_PATH=/usr/local/cuda-12.4/lib64:/usr/lib/x86_64-linux-gnu:/lib/x86_64-linux-gnu:/usr/lib:/lib --config=cuda_clang --action_env CLANG_CUDA_COMPILER_PATH=/usr/lib/llvm-17/bin/clang --copt=-Wno-gnu-offsetof-extensions --config=cuda_clang
INFO: Found applicable config definition build:short_logs in file /home/manveer/Downloads/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /home/manveer/Downloads/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:tensorrt in file /home/manveer/Downloads/tensorflow/.bazelrc: --repo_env TF_NEED_TENSORRT=1
INFO: Found applicable config definition build:cuda_clang in file /home/manveer/Downloads/tensorflow/.bazelrc: --config=cuda --config=tensorrt --action_env=TF_CUDA_CLANG=1 --@local_config_cuda//:cuda_compiler=clang --repo_env=TF_CUDA_COMPUTE_CAPABILITIES=sm_50,sm_60,sm_70,sm_80,compute_90
INFO: Found applicable config definition build:cuda in file /home/manveer/Downloads/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
INFO: Found applicable config definition build:tensorrt in file /home/manveer/Downloads/tensorflow/.bazelrc: --repo_env TF_NEED_TENSORRT=1
INFO: Found applicable config definition build:cuda_clang in file /home/manveer/Downloads/tensorflow/.bazelrc: --config=cuda --config=tensorrt --action_env=TF_CUDA_CLANG=1 --@local_config_cuda//:cuda_compiler=clang --repo_env=TF_CUDA_COMPUTE_CAPABILITIES=sm_50,sm_60,sm_70,sm_80,compute_90
INFO: Found applicable config definition build:cuda in file /home/manveer/Downloads/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
INFO: Found applicable config definition build:tensorrt in file /home/manveer/Downloads/tensorflow/.bazelrc: --repo_env TF_NEED_TENSORRT=1
INFO: Found applicable config definition build:opt in file /home/manveer/Downloads/tensorflow/.tf_configure.bazelrc: --copt=-Wno-sign-compare --host_copt=-Wno-sign-compare
INFO: Found applicable config definition build:linux in file /home/manveer/Downloads/tensorflow/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --copt=-Wno-error=unused-but-set-variable --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --experimental_guard_against_concurrent_changes
INFO: Found applicable config definition build:dynamic_kernels in file /home/manveer/Downloads/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
WARNING: The following configs were expanded more than once: [tensorrt, cuda_clang, cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
INFO: Build options --@local_config_cuda//:cuda_compiler, --action_env, --copt, and 2 more have changed, discarding analysis cache.
INFO: Analyzed target //tensorflow/tools/pip_package:build_pip_package (710 packages loaded, 51636 targets configured).
INFO: Found 1 target...
ERROR: /home/manveer/.cache/bazel/_bazel_manveer/795c5c27cbf66ab1777c71686be75148/external/local_config_tensorrt/BUILD:69:8: Executing genrule @local_config_tensorrt//:tensorrt_include failed: (Exit 1): bash failed: error executing command (from target @local_config_tensorrt//:tensorrt_include) /bin/bash -c ... (remaining 1 argument skipped)
cp: cannot stat '/usr/include/x86_64-linux-gnu/NvUtils.h': No such file or directory
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /home/manveer/Downloads/tensorflow/tensorflow/core/common_runtime/BUILD:79:23 Middleman _middlemen/_S_Stensorflow_Score_Scommon_Uruntime_Ccore_Ucpu_Uheaders_Ulib-cc_library-compile failed: (Exit 1): bash failed: error executing command (from target @local_config_tensorrt//:tensorrt_include) /bin/bash -c ... (remaining 1 argument skipped)
INFO: Elapsed time: 3.792s, Critical Path: 0.24s
INFO: 74 processes: 40 internal, 34 local.
FAILED: Build did NOT complete successfully
"""
```
