Unable to build TensorFlow without TensorRT
@torotoki 
During the configuration stage, explicitly disable TensorRT using the following command:

Bash
```
./configure  --no_tensorrt
```
Please double-check the generated .config file to ensure TensorRT is indeed disabled.
Thank you! @sushreebarsa Thank you for your response. But I couldn't find that argument or any related arguments. 
```
>>> ./configure --no_tensorrt
usage: configure.py [-h] [--workspace WORKSPACE]
configure.py: error: unrecognized arguments: --no_tensorrt
```

Also I couldn't find any `.config` file. (You meant `.bazel`?). Anyway, it is too hard to disable TensorRT in `.bazel` because there are so many lines that contains `tensorrt` which are incorporated in many other complicated building commands.

In my opinion, this is a bug related to the build code I can confirm of having the same problem, the only difference is I am using CUDA 12.2  and cuDNN 8.9.
Using either CLang v16 or v17 produces the same issue with tensorrt.
Also have tried some combinations of editing the Bazel Config file ".tf_configure.bazelrc" for -no_tensorrt, but still have the same result.

And did try of installing TensorRT without the purpose the using it, to see if the error would then be corrected, but the error remained. Just to add, taking a shot that perhaps the checkout version of 2.15.0 does not contain the Hotfix as indicated from here:
https://blog.tensorflow.org/2023/12/tensorflow-215-update-hot-fix-linux-installation-issue.html
According to the blog post, to install v2.15.0 is to use "2.15.0.post1", which does not appear to be possible with a checkout. Another update, interestingly I was able to compile from the Master (v2.16.0), but only by using GCC, as both versions of CLang v16 and v17 failed. I am however, unable to compile v2.15.0 with CLang or GCC.

I have to go through the process again as my current system is dirty from attempting to get TensorFlow with GPU to compile.  I had installed TensorRT with the intent of tricking v2.15.0 to compile without using TensorRt if that makes since. So I am not sure as if that was needed for v2.16.0 to compile. 

Used:
- Ubuntu v24.04 with updates/upgrades
- Base OS build of Python v3.10
- Nvidia driver v535
- Cuda v12.2
- cuDNN v8.9.0

Also interesting that the finished whl file requires Python v3.11 for use, but not to compile. 

So from what I gather, TensorFlow v2.15.0 is not possible to compile from source as per documentation. 
 Hello!
Another way to build from source is to use python3.10 with TensorFlow v2.14 which seems more stable.

- check out another release branch
`git checkout r2.14`
-  Run the configuration script
`./configure`
- Use **nvcc** instead of Clang to compile cuda.
> Do you want to use Clang to build TensorFlow? [Y/n] **N**


Continue with the instructions as described [here](https://www.tensorflow.org/install/source#configure_the_build).

I hope this helps. I do agree and for a stable version, though that does prevent the use of CUDA v12.
Below are the steps for building from source for the current version, v2.16.  I haven't put v2.16 into rigorous testing to determine its issues or limitations, however.
Although I have CLang listed below for how to install, I could only get TensorFlow v2.16 GPU to work with NVCC/GCC (as stated above for v2.14, by TheophileH)

**INSTALLING TENSORFLOW FOR GPU v2.16.0**

1. INSTALLATION UBUTUN DESKTOP v24.04.3 LTS
Used Minimal Installation
< Did not download 3rd Party Applications
< Ubuntu v22.04.3 LTS (Minimal Installation)
< Python3: v3.10.12
< No Nvida, Cuda, GCC, No CLang, No Bazel

2. PERFORM UPDATES
After installing Ubuntu, go to the Software Updater:
< Ubuntu Software > Download from: Main Server
< Click on Close
< Click on Reload
Open Software Updater:
< Additional Drivers > Select: Using NVIDIA driver metapackage from nvidia-driver-535 (proprietary,tested)
< Click on Apply Changes
< Close
Open Software Updater:
< Wait for window popup to perform updates, or click on the Nortification error due to Ubuntu Pro not loading, then click on Show Updates
< At the Software Updater, click on Install Now
< Click on Restart Now
< Allow Ubuntu to bootup normally this time

3. SETUP ENVIRONMENT
Open Terminal from Menu
To check for Nvidia Cards:
`lspci | grep -i nvidia`
To check the Nvidia Version:
`nvidia-smi | grep "Driver Version" | awk '{print $6}' | cut -c1-`
OR
`nvidia-smi`
Setup Python, can first check the current version:
`python3 -V`
< Result would be: Python 3.10.12
Add in Python Repositories and Dependencies
`sudo add-apt-repository ppa:ubuntu-toolchain-r/test`
`sudo add-apt-repository ppa:deadsnakes/ppa`
`sudo apt update -y && sudo apt upgrade -y`
`sudo apt install git python-is-python3 python3-pip python3-dev patchelf -y`
To Verify:
`python -V`
Set Path:
`sudo nano ~/.bashrc`
Add at the end of the file:
`export PATH="$PATH:/home/sysop/.local/bin"`
Save and exit
To apply changes:
`source ~/.bashrc`

4. INSTALL PYTHON v3.11.7:
To verify Python 3.11 is available:
`apt list | grep python3.11`
To install all Python 3.11 modules:
`sudo apt install python3.11-full -y`
Verify Install:
`python3.11 -V`
Result would be: Python 3.11.7
Set Alternative Versions for Python3:
`sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1`
`sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.10 2`
Python v3.10 is automatically set to Auto Mode, but can be edit as Manual:
`sudo update-alternatives --config python3`
< Select v3.10

5. INSTALL BAZELISK:
`cd Downloads`
`wget https://github.com/bazelbuild/bazelisk/releases/download/v1.19.0/bazelisk-linux-amd64`
`chmod +x bazelisk-linux-amd64`
`sudo mv bazelisk-linux-amd64 /usr/local/bin/bazel`
`sudo nano ~/.bashrc`
Add at the end:
`export PATH=/usr/local/bin/bazel:$PATH`
After closing the editor, type,
`source ~/.bashrc`

6. INSTALL CLANG v16:
`wget https://apt.llvm.org/llvm.sh`
`chmod +x llvm.sh`
`sudo ./llvm.sh 16`

OPTIONAL: IF DUAL MONITORS ARE NOT WORKING:
`sudo prime-select nvidia`
`sudo reboot`

7. PREPARE FOR TENSORFLOW:
Open Terminal from Menu
`pip install -U --user pip numpy wheel packaging requests opt_einsum`
`pip install -U --user keras_preprocessing --no-deps`
**I have found this trick to work, to get additional dependencies and of the correct versions:**
`pip install tensorflow==2.15.0 --upgrade`

8. PULL REPOSITORY TENSORFLOW:
`git clone https://github.com/tensorflow/tensorflow.git`
`cd tensorflow/`
**No Checkout**
Check for CPU Flags:
`grep flags -m1 /proc/cpuinfo | cut -d ":" -f 2 | tr '[:upper:]' '[:lower:]' | { read FLAGS; OPT="-march=native"; for flag in $FLAGS; do case "$flag" in "sse4_1" | "sse4_2" | "ssse3" | "fma" | "cx16" | "popcnt" | "avx" | "avx2") OPT+=" -m$flag";; esac; done; MODOPT=${OPT//_/\.}; echo "$MODOPT"; }`
< Copy the flags, such as:
`-march=native -mssse3 -mcx16 -msse4.1 -msse4.2 -mpopcnt`

9. NVIDIA CUDA TOOLKIT:
FOR CUDA v12.2:
Firefox: https://developer.nvidia.com/cuda-downloads
< Scroll to bottom, click on "Archive of Previous CUDA Releases"
< Click on: CUDA Toolkit 12.2
`cd Downloads`
`wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin`
`sudo mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600`
`wget https://developer.download.nvidia.com/compute/cuda/12.2.0/local_installers/cuda-repo-ubuntu2204-12-2-local_12.2.0-535.54.03-1_amd64.deb`
`sudo dpkg -i cuda-repo-ubuntu2204-12-2-local_12.2.0-535.54.03-1_amd64.deb`
`sudo cp /var/cuda-repo-ubuntu2204-12-2-local/cuda-*-keyring.gpg /usr/share/keyrings/`
`sudo apt update -y && sudo apt upgrade -y`
`sudo apt install cuda -y`
`sudo apt install nvidia-gds -y`
`sudo reboot`

10. ADD PERSISTENCE:
Log back in, open Terminal and type:
`export PATH=/usr/local/cuda-12.2/bin${PATH:+:${PATH}}`
`export LD_LIBRARY_PATH=/usr/local/cuda-12.2/lib64\
                         ${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}`
`source ~/.bashrc`
`systemctl status nvidia-persistenced`
Verify version:
`cat /proc/driver/nvidia/version`
Check Versions:
`nvidia-smi`
OR
`nvcc  --version`

11. INSTALL NVIDIA cuDNN:
FOR cuDNN v8.9.0:
(Requires registration/login)
Firefox: https://developer.nvidia.com/cudnn
Click on: Download cuDNN Library
Checkbox "I Agree To the Terms of the cuDNN Software License Agreement"
Click on the Archived cuDNN Releases link
Click on "Download cuDNN v8.9.0 (April 11th, 2023), for CUDA 12.x"
Click on "Local Installer for Ubuntu22.04 x86_64 (Deb)"
At Terminal:
`cd Downloads`
`sudo dpkg -i cudnn-local-repo-ubuntu2204-8.9.0.131_1.0-1_amd64.deb`
`sudo cp /var/cudnn-local-repo-*/cudnn-local-*-keyring.gpg /usr/share/keyrings/`
`sudo apt update -y`
`sudo apt install libcudnn8=8.9.0.131-1+cuda12.1`
`sudo apt install libcudnn8-samples=8.9.0.131-1+cuda12.1`
`sudo apt install make libfreeimage3 libfreeimage-dev`
Add CUDA Paths:
`echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc`
`echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc`
`source ~/.bashrc`
For Checking with Nvidia Settings:
`nvidia-smi`

As of Now: Ubuntu v22.04.3 LTS (Minimal Installation)
- Python3 v3.10.12
- Python3.11 v3.11.7
- Nvidia v535.129.03
- Cuda v12.2
- GCC v11.4.0
- CLang v16.06
- Bazel Bazelisk
- cuDNN: v8.9.0

12. CONFIGURE TENSORFLOW v2.16:
```
./configure
- Python Location:		Default (python3)
- Python Library:		Default (python3)
- Tensorflow with ROCm:		N
- Tensorflow with CUDA:		Y
- Tensorflow with TensorRT:	N
- CUDA Capabilities: 		Default
- CLang as CUDA Compiler: 	N
- GCC Path: 			Default
- Optimization Flags: 		-march=native -mssse3 -mcx16 -msse4.1 -msse4.2 -mpopcnt -Wno-gnu-offsetof-extensions
- Android Builds: 		N
```
To edit the Bazel Configuration because of an added use of "-Wno-gnu-offsetof-extensions":
`sudo nano .tf_configure.bazelrc`
Remove the first line entry of "-Wno-gnu-offsetof-extensions". Leave the remaining two lines in place (towards 2/3rds the way down)

10. COMPILE TENSORFLOW:
Part 1 - Build the package-builder:
`export TF_PYTHON_VERSION=3.10`
`sudo bazel build --config=opt --jobs=8 //tensorflow/tools/pip_package:build_pip_package`
Part 2 - Build the package:
`sudo ./bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg`

11. INSTALL TENSORFLOW:
Remove the existing Tensorflow v2.15
`cd ..`
`pip uninstall tensorflow`
Install the Tensorflow v2.16:
`python3.11 -m pip install /tmp/tensorflow_pkg/tensorflow*.whl --force-reinstall`

11. VERIFY TENSORFLOW:
`python3.11`
`import tensorflow as tf`
`print("Number of GPUs Available: ", len(tf.config.list_physical_devices('GPU')))`

END Same issue with TF 2.16.1

### Issue type

Build/Install

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

source

### TensorFlow version

nightly

### Custom code

Yes

### OS platform and distribution

Linux Ubuntu 22.04

### Mobile device

_No response_

### Python version

3.11

### Bazel version

6.1.0

### GCC/compiler version

clang-16

### CUDA/cuDNN version

CUDA 12.0.1/cuDNN 8.8

### GPU model and memory

GTX 1060

### Current behavior?

I followed the [tutorial](https://www.tensorflow.org/install/source) to build the TensorFlow nightly source code, but it aborted and asserted an error related to TensorRT, even though I had configured it to build TensorFlow *without* TensorRT.

Please see the attached Dockerfile for reproduction steps.
Then, I executed:
```
>>> cd tensorflow
>>> ./configure
You have bazel 6.1.0 installed.
Please specify the location of python. [Default is /usr/bin/python3]:


Found possible Python library paths:
  /usr/lib/python3/dist-packages
  /usr/local/lib/python3.10/dist-packages
Please input the desired Python library path to use.  Default is [/usr/lib/python3/dist-packages]

Do you wish to build TensorFlow with ROCm support? [y/N]: N
No ROCm support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: y
CUDA support will be enabled for TensorFlow.

Do you wish to build TensorFlow with TensorRT support? [y/N]: N
No TensorRT support will be enabled for TensorFlow.

Found CUDA 12.0 in:
    /usr/local/cuda-12.0/targets/x86_64-linux/lib
    /usr/local/cuda-12.0/targets/x86_64-linux/include
Found cuDNN 8 in:
    /usr/lib/x86_64-linux-gnu
    /usr/include


Please specify a list of comma-separated CUDA compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus. Each capability can be specified as "x.y" or "compute_xy" to include both virtual and binary GPU code, or as "sm_xy" to only include the binary code.
Please note that each additional compute capability significantly increases your build time and binary size, and that TensorFlow only supports compute capabilities >= 3.5 [Default is: 3.5,7.0]:


Do you want to use clang as CUDA compiler? [Y/n]: Y
Clang will be used as CUDA compiler.

Please specify clang path that to be used as host compiler. [Default is /usr/lib/llvm-16/bin/clang]:


You have Clang 16.0.6 installed.

Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -Wno-sign-compare]:


Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]:
Not configuring the WORKSPACE for Android builds.

Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See .bazelrc for more details.
        --config=mkl            # Build with MKL support.
        --config=mkl_aarch64    # Build with oneDNN and Compute Library for the Arm Architecture (ACL).
        --config=monolithic     # Config for mostly static monolithic build.
        --config=numa           # Build with NUMA support.
        --config=dynamic_kernels        # (Experimental) Build kernels into separate shared objects.
        --config=v1             # Build with TensorFlow 1 API instead of TF 2 API.
Preconfigured Bazel build configs to DISABLE default on features:
        --config=nogcp          # Disable GCP support.
        --config=nonccl         # Disable NVIDIA NCCL support.
Configuration finished
```
```
>>> bazel build --config=dbg //tensorflow/tools/pip_package:build_pip_package
ERROR: /tensorflow/WORKSPACE:84:14: fetching tensorrt_configure rule //external:local_config_tensorrt: Traceback (most recent call last):
        File "/tensorflow/third_party/tensorrt/tensorrt_configure.bzl", line 300, column 38, in _tensorrt_configure_impl
                _create_local_tensorrt_repository(repository_ctx)
        File "/tensorflow/third_party/tensorrt/tensorrt_configure.bzl", line 159, column 30, in _create_local_tensorrt_repository
                config = find_cuda_config(repository_ctx, ["cuda", "tensorrt"])
        File "/tensorflow/third_party/gpus/cuda_configure.bzl", line 652, column 26, in find_cuda_config
                exec_result = execute(repository_ctx, [python_bin, repository_ctx.attr._find_cuda_config] + cuda_libraries)
        File "/tensorflow/third_party/remote_config/common.bzl", line 230, column 13, in execute
                fail(
Error in fail: Repository command failed
Could not find any NvInferVersion.h matching version '' in any subdirectory:
        ''
        'include'
        'include/cuda'
        'include/*-linux-gnu'
        'extras/CUPTI/include'
        'include/cuda/CUPTI'
        'local/cuda/extras/CUPTI/include'
        'targets/x86_64-linux/include'
of:
        '/lib'
        '/lib32'
        '/usr'
        '/usr/lib/x86_64-linux-gnu'
        '/usr/local/cuda'
        '/usr/local/cuda/targets/x86_64-linux/lib'
ERROR: Skipping '//tensorflow/tools/pip_package:build_pip_package': no such package '@local_config_tensorrt//': Repository command failed
Could not find any NvInferVersion.h matching version '' in any subdirectory:
        ''
        'include'
        'include/cuda'
        'include/*-linux-gnu'
        'extras/CUPTI/include'
        'include/cuda/CUPTI'
        'local/cuda/extras/CUPTI/include'
        'targets/x86_64-linux/include'
of:
        '/lib'
        '/lib32'
        '/usr'
        '/usr/lib/x86_64-linux-gnu'
        '/usr/local/cuda'
        '/usr/local/cuda/targets/x86_64-linux/lib'
WARNING: Target pattern parsing failed.
ERROR: no such package '@local_config_tensorrt//': Repository command failed
Could not find any NvInferVersion.h matching version '' in any subdirectory:
        ''
        'include'
        'include/cuda'
        'include/*-linux-gnu'
        'extras/CUPTI/include'
        'include/cuda/CUPTI'
        'local/cuda/extras/CUPTI/include'
        'targets/x86_64-linux/include'
of:
        '/lib'
        '/lib32'
        '/usr'
        '/usr/lib/x86_64-linux-gnu'
        '/usr/local/cuda'
        '/usr/local/cuda/targets/x86_64-linux/lib'
INFO: Elapsed time: 0.171s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
    currently loading: tensorflow/tools/pip_package
```

In fact, the `NvInferVersion.h` file does not exist, but to my knowledge, `NvInferVersion.h` is supposed to be a header file for TensorRT. 

### Standalone code to reproduce the issue

See my Dockerfile that I reproduced the error on this:
https://gist.github.com/torotoki/8f88a4dd932c882018ccc3f38aab2ae7


### Relevant log output

_No response_