Cannot build tensorflow2.7-gpu version with bazel3.7.2
@xiaxia-wang,

I was able to clone the tensorflow repository without any problem on Ubuntu. I observed that you are using GCC 11, CUDA  12.1 and cuDNN 8.9 which is incompatible with TF v2.7.0.  And also TF v2.7.0 is a pretty older version, please try to install the latest stable version.

Could you please create a virtual environment and try to install the tensorflow as mentioned in this official document [link](https://www.tensorflow.org/install/source) and have a look at the [compatible](https://www.tensorflow.org/install/source#gpu) tested build configurations as well. Please find the attached screenshot for reference.

![Screenshot 2023-07-21 3 47 25 PM](https://github.com/tensorflow/tensorflow/assets/81610181/c06da998-2646-4810-afbc-3113d4f06aac)
 Thanks for your reply. 

I was able to clone the tensorflow repository without any problem. The checkout process also works well. My problem arises in the bazel build process. I explicitly follow every step mentioned in the official document [link](https://www.tensorflow.org/install/source), but when I run `bazel build --config=cuda //tensorflow/tools/pip_package:build_pip_package`, it turns out as the issue provided above. 

I have also read the table of compatible tests, but it shows the latest tested TF-GPU version is 2.6.0. Could you please explain more about the incompatibility? Is my tensorflow too old for the cuda version? If I remember it correctly, higher cuda version is compatible with lower requirements.

 @xiaxia-wang,
Is there any specific reason to install tensorflow v2.7, because as mentioned above v2.7 is the pretty older version.  It's unlikely for TF 2.7 version to receive any bug fixes except when we have security patches. There is a high possibility that this was fixed with later TF versions. 

We request please create a virtual environment and try to install the tensorflow latest stable v2.13 as mentioned in this official document [link](https://www.tensorflow.org/install/source) and have a look at the [compatible](https://www.tensorflow.org/install/source#gpu) tested build configurations as well. Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61345">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61345">No</a>
 Hi! i'm encountering the exact issue, may i ask have you resolved this? Unfortunately, no. They only told me to use newer versions of tensorflow...In the end I turned to pytorch and reconstructed my program, which for me is better and easier to use.
### Issue type

Build/Install

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

tf2.7

### Custom code

No

### OS platform and distribution

Linux Ubuntu 22.04

### Mobile device

Linux Ubuntu 22.04

### Python version

3.8

### Bazel version

3.7.2

### GCC/compiler version

gcc 11

### CUDA/cuDNN version

12.1/8.9

### GPU model and memory

Quadro RTX 8000, 48GB

### Current behavior?

Context: I can use the exact same source codes to successfully build the CPU-only version. I hope to get a gpu version with the same code.

So I ran `bazel clean`, it looks fine.

Then I run `bazel build --config=cuda //tensorflow/tools/pip_package:build_pip_package`. I have tried for several times, but everytime got the following error: (I only paste the main part here, before this there were some regular bazel output, e.g., INFO:...)

```
ERROR: An error occurred during the fetch of repository 'local_config_cuda':
   Traceback (most recent call last):
        File "/home/xiaxia/tensorflow/third_party/gpus/cuda_configure.bzl", line 1401, column 38, in _cuda_autoconf_impl
                _create_local_cuda_repository(repository_ctx)
        File "/home/xiaxia/tensorflow/third_party/gpus/cuda_configure.bzl", line 1076, column 27, in _create_local_cuda_repository
                cuda_libs = _find_libs(repository_ctx, check_cuda_libs_script, cuda_config)
        File "/home/xiaxia/tensorflow/third_party/gpus/cuda_configure.bzl", line 606, column 21, in _find_libs
                _check_cuda_libs(repository_ctx, check_cuda_libs_script, check_cuda_libs_params.values())
        File "/home/xiaxia/tensorflow/third_party/gpus/cuda_configure.bzl", line 501, column 28, in _check_cuda_libs
                checked_paths = execute(repository_ctx, [python_bin, "-c", cmd]).stdout.splitlines()
        File "/home/xiaxia/tensorflow/third_party/remote_config/common.bzl", line 230, column 13, in execute
                fail(
Error in fail: Repository command failed
Expected even number of arguments
INFO: Found applicable config definition build:cuda in file /home/xiaxia/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//c
INFO: Found applicable config definition build:cuda in file /home/xiaxia/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//c
WARNING: The following configs were expanded more than once: [cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
ERROR: @local_config_cuda//:enable_cuda :: Error loading option @local_config_cuda//:enable_cuda: Repository command failed
Expected even number of arguments
```

Note that, I do not have root/admin role of the remote ubuntu machine, will it be a problem? I also tried to implement cuda and cudnn in my /home/ directory, but it still not help.

Please provide me with any advice, and thanks a lot!

### Standalone code to reproduce the issue

```shell
Sorry I don't know how to share my problem as reproducible, it happens in the process of building tensorflow.
```


### Relevant log output

```shell
WARNING: The following configs were expanded more than once: [cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=157
INFO: Reading rc options for 'build' from /home/xiaxia/tensorflow/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /home/xiaxia/tensorflow/.bazelrc:
  'build' options: --define framework_shared_object=true --java_toolchain=@tf_toolchains//toolchains/java:tf_java_toolchain --host_java_toolchain=@tf_toolchains//toolchains/java:tf_java_toolchain --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true
INFO: Reading rc options for 'build' from /home/xiaxia/tensorflow/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=/home/xiaxia/anaconda3/envs/tf-cus-gpu/bin/python3 --action_env PYTHON_LIB_PATH=/home/xiaxia/anaconda3/envs/tf-cus-gpu/lib/python3.8/site-packages --python_path=/home/xiaxia/anaconda3/envs/tf-cus-gpu/bin/python3 --action_env TF_CUDA_VERSION=12.2 --action_env TF_CUDNN_VERSION=8 --action_env TF_NCCL_VERSION= --action_env TF_CUDA_PATHS=/home/xiaxia/cuda-12.2 --action_env CUDA_TOOLKIT_PATH=/home/xiaxia/cuda-12.2 --action_env TF_CUDA_COMPUTE_CAPABILITIES=3.5,7.0 --action_env LD_LIBRARY_PATH=/usr/local/cuda-12.1/targets/x86_64-linux/lib:/usr/local/cuda-12.1/targets/x86_64-linux/lib:/usr/local/cuda-12.1/targets/x86_64-linux/lib::/home/xiaxia/cuda-12.2/lib64:/home/xiaxia/cuda-12.2/lib64 --action_env GCC_HOST_COMPILER_PATH=/usr/bin/x86_64-linux-gnu-gcc-11 --config=cuda
INFO: Reading rc options for 'build' from /home/xiaxia/tensorflow/.bazelrc:
  'build' options: --deleted_packages=tensorflow/compiler/mlir/tfrt,tensorflow/compiler/mlir/tfrt/benchmarks,tensorflow/compiler/mlir/tfrt/jit/python_binding,tensorflow/compiler/mlir/tfrt/jit/transforms,tensorflow/compiler/mlir/tfrt/python_tests,tensorflow/compiler/mlir/tfrt/tests,tensorflow/compiler/mlir/tfrt/tests/saved_model,tensorflow/compiler/mlir/tfrt/transforms/lhlo_gpu_to_tfrt_gpu,tensorflow/core/runtime_fallback,tensorflow/core/runtime_fallback/conversion,tensorflow/core/runtime_fallback/kernel,tensorflow/core/runtime_fallback/opdefs,tensorflow/core/runtime_fallback/runtime,tensorflow/core/runtime_fallback/util,tensorflow/core/tfrt/common,tensorflow/core/tfrt/eager,tensorflow/core/tfrt/eager/backends/cpu,tensorflow/core/tfrt/eager/backends/gpu,tensorflow/core/tfrt/eager/core_runtime,tensorflow/core/tfrt/eager/cpp_tests/core_runtime,tensorflow/core/tfrt/fallback,tensorflow/core/tfrt/gpu,tensorflow/core/tfrt/run_handler_thread_pool,tensorflow/core/tfrt/runtime,tensorflow/core/tfrt/saved_model,tensorflow/core/tfrt/saved_model/tests,tensorflow/core/tfrt/tpu,tensorflow/core/tfrt/utils
INFO: Found applicable config definition build:short_logs in file /home/xiaxia/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /home/xiaxia/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:cuda in file /home/xiaxia/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
INFO: Found applicable config definition build:cuda in file /home/xiaxia/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
INFO: Found applicable config definition build:linux in file /home/xiaxia/tensorflow/.bazelrc: --copt=-w --host_copt=-w --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++14 --host_cxxopt=-std=c++14 --config=dynamic_kernels --distinct_host_configuration=false --experimental_guard_against_concurrent_changes
INFO: Found applicable config definition build:dynamic_kernels in file /home/xiaxia/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
INFO: Repository local_config_cuda instantiated at:
  /home/xiaxia/tensorflow/WORKSPACE:15:14: in <toplevel>
  /home/xiaxia/tensorflow/tensorflow/workspace2.bzl:1080:19: in workspace
  /home/xiaxia/tensorflow/tensorflow/workspace2.bzl:94:19: in _tf_toolchains
Repository rule cuda_configure defined at:
  /home/xiaxia/tensorflow/third_party/gpus/cuda_configure.bzl:1448:33: in <toplevel>
ERROR: An error occurred during the fetch of repository 'local_config_cuda':
   Traceback (most recent call last):
        File "/home/xiaxia/tensorflow/third_party/gpus/cuda_configure.bzl", line 1401, column 38, in _cuda_autoconf_impl
                _create_local_cuda_repository(repository_ctx)
        File "/home/xiaxia/tensorflow/third_party/gpus/cuda_configure.bzl", line 1076, column 27, in _create_local_cuda_repository
                cuda_libs = _find_libs(repository_ctx, check_cuda_libs_script, cuda_config)
        File "/home/xiaxia/tensorflow/third_party/gpus/cuda_configure.bzl", line 606, column 21, in _find_libs
                _check_cuda_libs(repository_ctx, check_cuda_libs_script, check_cuda_libs_params.values())
        File "/home/xiaxia/tensorflow/third_party/gpus/cuda_configure.bzl", line 501, column 28, in _check_cuda_libs
                checked_paths = execute(repository_ctx, [python_bin, "-c", cmd]).stdout.splitlines()
        File "/home/xiaxia/tensorflow/third_party/remote_config/common.bzl", line 230, column 13, in execute
                fail(
Error in fail: Repository command failed
Expected even number of arguments
INFO: Found applicable config definition build:cuda in file /home/xiaxia/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
INFO: Found applicable config definition build:cuda in file /home/xiaxia/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
WARNING: The following configs were expanded more than once: [cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
ERROR: @local_config_cuda//:enable_cuda :: Error loading option @local_config_cuda//:enable_cuda: Repository command failed
Expected even number of arguments
```
