Errors when building v2.14 from source - ubuntu 20.04 / arm64 / GPU + CUDA (capabilities 8.7) / clang 16
Hi @antoniaelsen ,

By looking at the logs it seems your environment have CUDA 11.4 and cuDNN 8 versions which may raise compatibility issues. For TF2.14v, CUDA=11.8 and cuDNN=8.7 are tested versions.

Please refer the official tested build configurations [here](https://www.tensorflow.org/install/source#gpu).

Could you test with official build configurations and let us know the outcome.

Thanks! > Hi @antoniaelsen ,
> 
> By looking at the logs it seems your environment have CUDA 11.4 and cuDNN 8 versions which may raise compatibility issues. For TF2.14v, CUDA=11.8 and cuDNN=8.7 are tested versions.
> 
> Please refer the official tested build configurations [here](https://www.tensorflow.org/install/source#gpu).
> 
> Could you test with official build configurations and let us know the outcome.
> 
> Thanks!

That did it, thanks :) Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62109">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62109">No</a>

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

v2.14.0

### Custom code

No

### OS platform and distribution

Ubuntu 20.04.6

### Mobile device

_No response_

### Python version

3.10.13

### Bazel version

6.1.0

### GCC/compiler version

clang 16.0.6

### CUDA/cuDNN version

11.4 / 8.6.0

### GPU model and memory

jetson orin agx  -> nvidia ampere

### Current behavior?

Attempting to build TF from source
- Jetson Orin w Ampere GPU, arm64
- TF 2.14.0
- bazel 6.1.0
- python 3.10

`./configure`
- no ROCm
- yes CUDA (capabilities = 8.7)
- no TensorRT
- using clang 16 as cuda compiler

`bazel build //tensorflow/tools/pip_package:build_pip_package`

Resultng behavior: build fails, with the following error:
```
.../tensorflow/core/kernels/BUILD:4996:18: Compiling tensorflow/core/kernels/sparse_reorder_op_gpu.cu.cc failed: (Exit 1): clang failed: error executing command (from target
 //tensorflow/core/kernels:sparse_reorder_op_gpu) /usr/lib/llvm-16/bin/clang -MD -MF bazel-out/aarch64-opt/bin/tensorflow/core/kernels/_objs/sparse_reorder_op_gpu/sparse_reorder_op_gpu.cu.pic.d ... (remaining 192 a
rguments skipped)
In file included from tensorflow/core/kernels/sparse_reorder_op_gpu.cu.cc:21:
In file included from ./tensorflow/core/kernels/gpu_prim.h:24:
In file included from bazel-out/aarch64-opt/bin/external/local_config_cuda/cuda/cuda/include/cub/device/device_radix_sort.cuh:40:
In file included from bazel-out/aarch64-opt/bin/external/local_config_cuda/cuda/cuda/include/cub/device/dispatch/dispatch_radix_sort.cuh:40:
In file included from bazel-out/aarch64-opt/bin/external/local_config_cuda/cuda/cuda/include/cub/device/dispatch/../../agent/agent_radix_sort_histogram.cuh:38:
bazel-out/aarch64-opt/bin/external/local_config_cuda/cuda/cuda/include/cub/device/dispatch/../../agent/../block/radix_rank_sort_operations.cuh:124:20: error: explicit qualification required to use member 'ProcessFl
oatMinusZero' from dependent base class
        return BFE(ProcessFloatMinusZero(key), bit_start, num_bits);
                   ^
 ...
 
 bazel-out/aarch64-opt/bin/external/local_config_cuda/cuda/cuda/include/cub/device/device_radix_sort.cuh:168:65: note: in instantiation of member function 'cub::DispatchRadixSort<false, long, long, int>::Dispatch' r
equested here
        return DispatchRadixSort<false, KeyT, ValueT, OffsetT>::Dispatch(
                                                                ^
./tensorflow/core/kernels/gpu_prim_helpers.h:117:37: note: in instantiation of function template specialization 'cub::DeviceRadixSort::SortPairs<long, long>' requested here
    err = gpuprim::DeviceRadixSort::SortPairs(
                                    ^
./tensorflow/core/kernels/gpu_prim_helpers.h:159:18: note: in instantiation of function template specialization 'tensorflow::detail::GpuRadixSortImpl<false, long, long>' requested here
  return detail::GpuRadixSortImpl</*Descending=*/false>(
                 ^
tensorflow/core/kernels/sparse_reorder_op_gpu.cu.cc:104:12: note: in instantiation of function template specialization 'tensorflow::GpuRadixSort<long, long>' requested here
        c, GpuRadixSort(c, num_elems, /*keys_in=*/flat_indices,
           ^
bazel-out/aarch64-opt/bin/external/local_config_cuda/cuda/cuda/include/cub/device/dispatch/../../agent/../block/radix_rank_sort_operations.cuh:98:52: note: member is declared here
    static __device__ __forceinline__ UnsignedBits ProcessFloatMinusZero(UnsignedBits key)
                                                   ^
2 errors generated when compiling for sm_87.
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 9646.040s, Critical Path: 487.26s
INFO: 17934 processes: 6479 internal, 11455 local.
FAILED: Build did NOT complete successfully
```

### Standalone code to reproduce the issue

```shell
see below
```


### Relevant log output

```shell
<someuser>:tensorflow ➜  git status
HEAD detached at v2.14.0
nothing to commit, working tree clean
<someuser>:tensorflow ➜  ./configure
You have bazel 6.1.0 installed.
Please specify the location of python. [Default is /home/someuser/.pyenv/versions/3.10.13/bin/python3]:


Found possible Python library paths:
  /home/someuser/.pyenv/versions/3.10.13/lib/python3.10/site-packages
Please input the desired Python library path to use.  Default is [/home/someuser/.pyenv/versions/3.10.13/lib/python3.10/site-packages]

Do you wish to build TensorFlow with ROCm support? [y/N]:
No ROCm support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: y
CUDA support will be enabled for TensorFlow.

Do you wish to build TensorFlow with TensorRT support? [y/N]:
No TensorRT support will be enabled for TensorFlow.

Found CUDA 11.4 in:
    /usr/local/cuda-11.4/targets/aarch64-linux/lib
    /usr/local/cuda-11.4/targets/aarch64-linux/include
Found cuDNN 8 in:
    /usr/lib/aarch64-linux-gnu
    /usr/include


Please specify a list of comma-separated CUDA compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus. Each capability can be specified as "x.y" or "compute_xy" to include both virtual and binary GPU code, or as "sm_xy" to only include the binary code.
Please note that each additional compute capability significantly increases your build time and binary size, and that TensorFlow only supports compute capabilities >= 3.5 [Default is: 3.5,7.0]: 8.7


Do you want to use clang as CUDA compiler? [Y/n]:
Clang will be used as CUDA compiler.

Please specify clang path that to be used as host compiler. [Default is /usr/lib/llvm-16/bin/clang]:


You have Clang 16.0.6 installed.

Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -Wno-sign-compare]:


Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: N
Not configuring the WORKSPACE for Android builds.

Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See .bazelrc for more details.
        --config=mkl            # Build with MKL support.
        --config=mkl_aarch64    # Build with oneDNN and Compute Library for the Arm Architecture (ACL).
        --config=monolithic     # Config for mostly static monolithic build.
        --config=numa           # Build with NUMA support.
        --config=dynamic_kernels        # (Experimental) Build kernels into separate shared objects.
        --config=v1             # Build with TensorFlow 1 API instead of TF 2 API.
Preconfigured Bazel build configs to DISABLE default on features:
        --config=nogcp          # Disable GCP support.
        --config=nonccl         # Disable NVIDIA NCCL support.
Configuration finished
<someuser>:tensorflow ➜  bazel build //tensorflow/tools/pip_package:build_pip_package
WARNING: while reading option defaults file '/home/someuser/projects/other/tensorflow/.bazelrc':
  invalid command name 'startup:windows'.
WARNING: The following configs were expanded more than once: [cuda_clang, cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=214
INFO: Reading rc options for 'build' from /home/someuser/projects/other/tensorflow/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /home/someuser/projects/other/tensorflow/.bazelrc:
  'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --features=-force_no_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility
INFO: Reading rc options for 'build' from /home/someuser/projects/other/tensorflow/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=/home/someuser/.pyenv/versions/3.10.13/bin/python3 --action_env PYTHON_LIB_PATH=/home/someuser/.pyenv/versions/3.10.13/lib/python3.10/site-packages --python_path=/home/someuser/.pyenv/versions/3.10.13/bin/python3 --action_env CUDA_TOOLKIT_PATH=/usr/local/cuda-11.4 --action_env TF_CUDA_COMPUTE_CAPABILITIES=8.7 --config=cuda_clang --action_env CLANG_CUDA_COMPILER_PATH=/usr/lib/llvm-16/bin/clang --copt=-Wno-gnu-offsetof-extensions --config=cuda_clang
INFO: Found applicable config definition build:short_logs in file /home/someuser/projects/other/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /home/someuser/projects/other/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:cuda_clang in file /home/someuser/projects/other/tensorflow/.bazelrc: --config=cuda --repo_env TF_CUDA_CLANG=1 --@local_config_cuda//:cuda_compiler=clang
INFO: Found applicable config definition build:cuda in file /home/someuser/projects/other/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
INFO: Found applicable config definition build:cuda_clang in file /home/someuser/projects/other/tensorflow/.bazelrc: --config=cuda --repo_env TF_CUDA_CLANG=1 --@local_config_cuda//:cuda_compiler=clang
INFO: Found applicable config definition build:cuda in file /home/someuser/projects/other/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
INFO: Found applicable config definition build:linux in file /home/someuser/projects/other/tensorflow/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --copt=-Wno-error=unused-but-set-variable --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --experimental_guard_against_concurrent_changes
INFO: Found applicable config definition build:dynamic_kernels in file /home/someuser/projects/other/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/llvm/llvm-project/archive/668e33c6401abe7844691fb7d47a3cf2d2012dbc.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/tensorflow/runtime/archive/769f5cc9b8732933140b09e8808d13614182b496.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: while reading option defaults file '/home/someuser/projects/other/tensorflow/.bazelrc':
  invalid command name 'startup:windows'.
WARNING: The following configs were expanded more than once: [cuda_clang, cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
INFO: Build options --copt and --host_copt have changed, discarding analysis cache.
WARNING: Download from https://mirror.bazel.build/github.com/bazelbuild/rules_cc/archive/081771d4a0e9d7d3aa0eed2ef389fa4700dfb23e.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/openxla/stablehlo/archive/9ae6c373a6e2941ff84a8831bb3724728cb2b49a.zip failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/pytorch/cpuinfo/archive/87d8234510367db49a65535021af5e1838a65ac2.zip failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/google/XNNPACK/archive/b9d4073a6913891ce9cbd8965c8d506075d2a45a.zip failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/openxla/triton/archive/cl546794996.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
INFO: Analyzed target //tensorflow/tools/pip_package:build_pip_package (1 packages loaded, 47122 targets configured).
INFO: Found 1 target...
[1,765 / 6,027] 8 actions running
[1,802 / 6,027] 8 actions running
    Compiling mlir/lib/Dialect/Arith/IR/ArithOps.cpp [for tool]; 41s local
    Compiling mlir/lib/Dialect/Arith/IR/ArithDialect.cpp [for tool]; 40s local
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: build interrupted
INFO: Elapsed time: 195.807s, Critical Path: 101.10s
INFO: 301 processes: 15 internal, 286 local.
FAILED: Build did NOT complete successfully

<someuser>:tensorflow ➜
<someuser>:tensorflow ➜

<someuser>:tensorflow ➜
<someuser>:tensorflow ➜  reset

<someuser>:tensorflow ➜  bazel clean
WARNING: while reading option defaults file '/home/someuser/projects/other/tensorflow/.bazelrc':
  invalid command name 'startup:windows'.
WARNING: The following configs were expanded more than once: [cuda_clang, cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=214
INFO: Reading rc options for 'clean' from /home/someuser/projects/other/tensorflow/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'clean' from /home/someuser/projects/other/tensorflow/.bazelrc:
  Inherited 'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --features=-force_no_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility
INFO: Reading rc options for 'clean' from /home/someuser/projects/other/tensorflow/.tf_configure.bazelrc:
  Inherited 'build' options: --action_env PYTHON_BIN_PATH=/home/someuser/.pyenv/versions/3.10.13/bin/python3 --action_env PYTHON_LIB_PATH=/home/someuser/.pyenv/versions/3.10.13/lib/python3.10/site-packages --python_path=/home/someuser/.pyenv/versions/3.10.13/bin/python3 --action_env CUDA_TOOLKIT_PATH=/usr/local/cuda-11.4 --action_env TF_CUDA_COMPUTE_CAPABILITIES=8.7 --config=cuda_clang --action_env CLANG_CUDA_COMPILER_PATH=/usr/lib/llvm-16/bin/clang --copt=-Wno-gnu-offsetof-extensions --config=cuda_clang
INFO: Found applicable config definition build:short_logs in file /home/someuser/projects/other/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /home/someuser/projects/other/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:cuda_clang in file /home/someuser/projects/other/tensorflow/.bazelrc: --config=cuda --repo_env TF_CUDA_CLANG=1 --@local_config_cuda//:cuda_compiler=clang
INFO: Found applicable config definition build:cuda in file /home/someuser/projects/other/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
INFO: Found applicable config definition build:cuda_clang in file /home/someuser/projects/other/tensorflow/.bazelrc: --config=cuda --repo_env TF_CUDA_CLANG=1 --@local_config_cuda//:cuda_compiler=clang
INFO: Found applicable config definition build:cuda in file /home/someuser/projects/other/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
INFO: Found applicable config definition build:linux in file /home/someuser/projects/other/tensorflow/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --copt=-Wno-error=unused-but-set-variable --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --experimental_guard_against_concurrent_changes
INFO: Found applicable config definition build:dynamic_kernels in file /home/someuser/projects/other/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
<someuser>:tensorflow ➜  reset
'





















<someuser>:tensorflow ➜  git status
HEAD detached at v2.14.0
nothing to commit, working tree clean
<someuser>:tensorflow ➜  bazel --version
bazel 6.1.0
<someuser>:tensorflow ➜  clang-16 --version
Ubuntu clang version 16.0.6 (++20230710042046+7cbf1a259152-1~exp1~20230710162136.105)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
<someuser>:tensorflow ➜  ./configure
You have bazel 6.1.0 installed.
Please specify the location of python. [Default is /home/someuser/.pyenv/versions/3.10.13/bin/python3]:


Found possible Python library paths:
  /home/someuser/.pyenv/versions/3.10.13/lib/python3.10/site-packages
Please input the desired Python library path to use.  Default is [/home/someuser/.pyenv/versions/3.10.13/lib/python3.10/site-packages]

Do you wish to build TensorFlow with ROCm support? [y/N]:
No ROCm support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: y
CUDA support will be enabled for TensorFlow.

Do you wish to build TensorFlow with TensorRT support? [y/N]:
No TensorRT support will be enabled for TensorFlow.

Found CUDA 11.4 in:
    /usr/local/cuda-11.4/targets/aarch64-linux/lib
    /usr/local/cuda-11.4/targets/aarch64-linux/include
Found cuDNN 8 in:
    /usr/lib/aarch64-linux-gnu
    /usr/include


Please specify a list of comma-separated CUDA compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus. Each capability can be specified as "x.y" or "compute_xy" to include both virtual and binary GPU code, or as "sm_xy" to only include the binary code.
Please note that each additional compute capability significantly increases your build time and binary size, and that TensorFlow only supports compute capabilities >= 3.5 [Default is: 3.5,7.0]: 8.7


Do you want to use clang as CUDA compiler? [Y/n]:
Clang will be used as CUDA compiler.

Please specify clang path that to be used as host compiler. [Default is /usr/lib/llvm-16/bin/clang]:


You have Clang 16.0.6 installed.

Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -Wno-sign-compare]:


Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]:
Not configuring the WORKSPACE for Android builds.

Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See .bazelrc for more details.
        --config=mkl            # Build with MKL support.
        --config=mkl            # Build with MKL support.                                                                                                                                                    [12/7774]
        --config=mkl_aarch64    # Build with oneDNN and Compute Library for the Arm Architecture (ACL).
        --config=monolithic     # Config for mostly static monolithic build.
        --config=numa           # Build with NUMA support.
        --config=dynamic_kernels        # (Experimental) Build kernels into separate shared objects.
        --config=v1             # Build with TensorFlow 1 API instead of TF 2 API.
Preconfigured Bazel build configs to DISABLE default on features:
        --config=nogcp          # Disable GCP support.
        --config=nonccl         # Disable NVIDIA NCCL support.
Configuration finished
<someuser>:tensorflow ➜  bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package



/////////////////////////
```
