Build from source 2.13.0 fails on Rocky Linux 8.8
@nadyawilliams Could you please run bazel clean --expunge, forcing bazel to re-download and build each dependency to avoid issues. Also sync the bazel build bazel sync.
Thank you! Thank you for a suggestion.
I did bazel clean --expunge, before the build and cleaned again when i tried different options. No help.
Running bazel sync fails with multiple errors starting rom not finding cuda.h (cuda module is loaded and cuda.h is available in 
cuda install), followed by not finding python (also is loaded via module and available). Looks like bazel sync is not using any
of the variables set by the modules.  For example a python error is
ERROR: An error occurred during the fetch of repository 'ubuntu20.04-gcc9_manylinux2014-cuda11.8-cudnn8.6-tensorrt8.4_config_python3.11':
   Traceback (most recent call last):
	File "/export/repositories/tensorflow-admix/yamlspecs/tensorflow-2.13.0/third_party/py/python_configure.bzl", line 212, column 22, in _create_local_python_repository
		_check_python_bin(repository_ctx, python_bin)
	File "/export/repositories/tensorflow-admix/yamlspecs/tensorflow-2.13.0/third_party/py/python_configure.bzl", line 145, column 25, in _check_python_bin
		auto_config_fail("--define %s='%s' is not executable. Is it the python binary?" % (
	File "/export/repositories/tensorflow-admix/yamlspecs/tensorflow-2.13.0/third_party/remote_config/common.bzl", line 12, column 9, in auto_config_fail
		fail("%sConfiguration Error:%s %s\n" % (red, no_color, msg))
Error in fail: Configuration Error: --define PYTHON_BIN_PATH='/usr/local/bin/python3.11' is not executable. Is it the python binary?

Per my modules python is
which python
/opt/apps/python/3.10.2/bin/python
 Hello, @nadyawilliams! If you are using TF v2.13 then the supported version for CUDA and cuDNN are 11.8, 8.6 respectively.  Could you please verify the tested build configuration as mentioned [here](https://www.tensorflow.org/install/source#gpu) and let us know if that helps?
Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. I will be able to proceed with the testing of  cuda  and cudNN  11.8 and 8.6 the earliest next week. both build have to be done according to our build schedule and approach. Please keep the issue open till i try this.  Thank you Unfortunately at this time we cant upgrade our CUDA installation to 11.8. Please advise which  latest version of tensorflow can be compiled  with cuda 11.7 and cudNN 8.5. Thank you. @nadyawilliams Thanks for your response! We need to use the recommended compatible versions only otherwise the build issues would be hard to resolve. Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. Please close the issue. I  can compile tensorflow 2.11 with our current cuda install and while it is not the latest it will do for now. @nadyawilliams Thank you for the confirmation. 
Closing the ticket for now. Thank you! Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61978">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61978">No</a>

### Issue type

Build/Install

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

2.13.0

### Custom code

No

### OS platform and distribution

Rocky Linux 8.8

### Mobile device

_No response_

### Python version

3.10.2

### Bazel version

5.3.0

### GCC/compiler version

clang 16.0.1 or gcc 11.2.0

### CUDA/cuDNN version

cuda 11.7.1 /  cuDNN 8.5.0.96

### GPU model and memory

_No response_

### Current behavior?

1. load modules (all built locally on the host  and provide an access to the specified software) 
 **module load  bazel/5.3.0    gcc/11.2.0    llvm/16.0.1   clang/16.0.1  python/3.10.2   cuda/11.7.1   tensorRT/8.4.2.4**  

2.  Configure tensroflow:
**./configure**

Resulting  .tf_configure.bazelrc is:
build --action_env PYTHON_BIN_PATH="/opt/apps/python/3.10.2/bin/python3"
build --action_env PYTHON_LIB_PATH="/opt/apps/python/3.10.2/lib/python3.10/site-packages"
build --python_path="/opt/apps/python/3.10.2/bin/python3"
build --config=tensorrt
build --action_env TF_CUDA_VERSION="11"
build --action_env TF_CUDNN_VERSION="8"
build --action_env TF_TENSORRT_VERSION="8"
build --action_env TF_NCCL_VERSION=""
build --action_env TF_CUDA_PATHS="/opt/apps/cuda/11.7.1,/opt/apps/tensorRT/8.4.2.4,/usr"
build --action_env CUDA_TOOLKIT_PATH="/opt/apps/cuda/11.7.1"
build --action_env TF_CUDA_COMPUTE_CAPABILITIES="7.0"
build --action_env LD_LIBRARY_PATH="/opt/apps/tensorRT/8.4.2.4/lib:/opt/apps/cuda/11.7.1/lib64:/opt/apps/cuda/11.7.1/nvvm/lib64:/opt/apps/cuda/11.7.1/cublas/lib64:/opt/apps/cuda/11.7.1/extras/CUPTI/lib64:/opt/apps/cuda/11.7.1
/extras/Debugger/lib64:/opt/apps/python/3.10.2/lib:/opt/apps/clang/16.0.1/lib:/opt/apps/llvm/16.0.1/lib:/opt/apps/gcc/11.2.0/lib64:/opt/apps/gcc/11.2.0/lib:/opt/apps/gcc/11.2.0/lib/gcc/x86_64-pc-linux-gnu/11.2.0"
build --config=cuda_clang
build --action_env CLANG_CUDA_COMPILER_PATH="/opt/apps/clang/16.0.1/bin/clang"
build --config=cuda_clang
build:opt --copt=-mavx2
build:opt --host_copt=-mavx2
test --flaky_test_attempts=3
test --test_size_filters=small,medium
test --test_env=LD_LIBRARY_PATH
test:v1 --test_tag_filters=-benchmark-test,-no_oss,-oss_excluded,-no_gpu,-oss_serial
test:v1 --build_tag_filters=-benchmark-test,-no_oss,-oss_excluded,-no_gpu
test:v2 --test_tag_filters=-benchmark-test,-no_oss,-oss_excluded,-no_gpu,-oss_serial,-v1only
test:v2 --build_tag_filters=-benchmark-test,-no_oss,-oss_excluded,-no_gpu,-v1only

3 Run bazel build
**nohup bazel build --config=opt --jobs=8 --verbose_failures --verbose_explanations \
            --explain=/tmp/explain.txt //tensorflow/tools/pip_package:build_pip_package > build-out.txt &**

The build fails with an error. Attaching output log  files build-out.txt (collect errors) and explain.txt ( collect verbose output)
per above command:
[build-out.txt](https://github.com/tensorflow/tensorflow/files/12719249/build-out.txt)
[explain.txt](https://github.com/tensorflow/tensorflow/files/12719259/explain.txt)

If i compile without cuda/tensorRT, the clang compiler is not chosen and there is no way to choose is per current configure.py
as clang choose seem to be used only for gpu-enabled builds. In this case, gcc 11.2.0 is used and the error happens in a different area.

I tried to use different options adding **--config=cuda** or **--cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0"** and every 
time i get  a failure on a different file yet always  the culprit seem to be these lines from the  respective build output:

In file included from /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/memory:77:
In file included from /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/shared_ptr.h:53:
/opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/shared_ptr_base.h:196:22: error: type name does not allow function specifier to be specified
bazel-out/k8-opt/bin/external/local_config_cuda/cuda/cuda/include/crt/host_defines.h:83:24: note: expanded from macro '__noinline__'
        __attribute__((noinline))

Note, i dont have **/opt/rh...** path on this host so this must  be coming during compiling some external  dependencies 
and looking at my .tf_configure.bazelrc file i dont  really have control over how external dependencies are handled. 
Can reproduce with nightly builds as it uses different  (newer) version of tensorflow and  bazel.

I have previously compiled tensorflow 2.8.0 (with  its corresponding bazel version)  on the same host with the same cuda, python, and gcc using identical build  steps and the builds went perfectly file. I am confident that all prerequisite software is installed correctly (using it for many other packages builds outside tensorflow).

I would really appreciate if someone can point to what am  i doing wrong or what else i can try. 

No conda or docker please. Neither one will work in our environment where i need home built RPMS 
for installation on the HPC cluster. 

Thanks!

### Standalone code to reproduce the issue

```shell
No test case
```


### Relevant log output

_No response_