crosstool_wrapper_driver_is_not_gcc error while building TF v2.11
Hi @predrag12, please let us know if you are trying to build Tensorflow in Docker. Thank you!
   Hi, yes, container name is under exact commands to reproduce. @predrag12,
Could you please confirm whether you are trying to install from the official tensorflow docker image
https://hub.docker.com/r/tensorflow/tensorflow/
If No, I suggest you to use one of the tested build configurations (please check versions):

https://www.tensorflow.org/install/source#tested_build_configurations. 
Also Please let us know after testing with tested configs. ( GCC 9.3.1 , Bazel 5.3.0, cuDNN 8.1, CUDA 11.2, 2.11) and the reference [issue](https://github.com/tensorflow/tensorflow/issues/58419). Thank you!
 Hi, official tensorflow docker images do not contain any aarch64 tags.

Building using gcc 9.3 Bazel 5.3 just newer cuDNN 8.7 Cuda 11.8 listed in the problem description.

Based on nvcr.io/nvidia/tensorflow:22.12-tf2-py3, 8.7 and 11.8 are compatible with 2.10.1, but build based on https://www.tensorflow.org/install/source instructions fails with error above.

Based on referenced issue #58419 Cuda 11.8 is compatible once TensorRT is turned off, which is also the case here (off), but build still fails with said error above.

Building within nvidia/cuda:11.2.2-cudnn8-devel-ubuntu20.04 (cudNN 8.1 Cuda 11.2) fails with even worse errors, below. Thanks.


ERROR: /root/.cache/bazel/_bazel_root/68a62076e91007a7908bc42a32e4cff9/external/local_config_python/BUILD:78:8: in outs attribute of genrule rule @local_config_python//:python_include: Genrules without outputs don't make sense

ERROR: /root/.cache/bazel/_bazel_root/68a62076e91007a7908bc42a32e4cff9/external/local_config_python/BUILD:78:8: Analysis of target '@local_config_python//:python_include' failed

ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' failed; build aborted: 
FAILED: Build did NOT complete successfully (391 packages loaded, 5912 targets configured) Hi @predrag12 ,

Please check the version compatibility w.r.t CUDA and cuDNN.Supported configurations are CUDA=11.2 and cuDNN=8.1. Though for the referenced issue it may worked but Architecture may be different and compatibility issues may arise.

While installing bazel did you covered all the 3-steps mentioned [here](https://bazel.build/install/ubuntu#install-packages).

Also check the compatibility of g++ version w.r.t gcc version in your environment and ensure compatible versions installed.

Please refer to this command [here](https://github.com/tensorflow/tensorflow/issues/44692#issuecomment-917543931) for similar problem here some resource links here [link1](https://github.com/tensorflow/serving/issues/976) & [link2](https://askubuntu.com/questions/749441/tensorflow-crosstool-wrapper-driver-is-not-gcc-failed-error-executing-command) and check whether they are of any help.

After checking all above mentioned checks please let us know the result. Thankyou! Hi,

For nvidia/cuda:11.2.2-cudnn8-devel-ubuntu20.04 (cudNN **8.1** Cuda **11.2**) there are errors at the onset of the compilation during configuration. Downgrading to gcc to 9.3.0, 9.2.1 doesn't change the outcome.

Installation of bazel 5.3.0 for aarch64 is different then those steps, but it works in nvidia/cuda:11.8.0-cudnn8-devel-ubuntu20.04, at least until linking.

Based on all the crosstool_wrapper_driver_is_not_gcc links above, it looks like there are many causes for the same reported error, most were not resolved, and likely different cause since x86_64 build works.

Would you be able to provide specific container tag and build commands that that work for aarch, even though might be different from ones in https://www.tensorflow.org/install/source that do not work? Based on nvcr.io/nvidia/tensorflow:22.12-tf2-py3, TF 2.10.1 is buildable, just that instructions are not documented either on NV or TF site? Thanks. @predrag12 ,

AFAIK, We don't have separate instructions for Arm/AArch builds. As you mentioned those earlier with TF2.8v the same instructions worked we need to look whether this is a regression issue.Could you please confirm the last TF versions it worked well.

Also I am trying to access the instructions at `nvcr.io/nvidia/tensorflow:22.12-tf2-py3` which I don't have access.

If it is OK to provide the instructions mentioned there, I may cross check it definitely and might have some useful info there which helps us in debugging this issue.

 Hi,
Don't recall if it was 2.8 or 2.4 or earlier and earlier Cuda.

If you docker run nvidia/cuda:11.2.2-cudnn8-devel-ubuntu20.04 for arrch64 and execute pip install tensorflow, it lists 2.10.0rc0 to 2.12.0 including 2.11.0, so question is how is it being build for aarch64 if official TF instructions don't work and there are no separate TF instructions for aarch64?

I don't have any special access to nvcr.io/nvidia/tensorflow:22.12-tf2-py3, it is a public docker image for aarch64 which you can get with docker pull and could look at its image history.

Thanks. Confirmed that **2.8.0** builds with bazel 4.2.1 within nvidia/cuda:**11.2**.2-cudnn**8**-devel-ubuntu20.04. So build with https://www.tensorflow.org/install/source instructions was broken sometime between 2.8.1 and 2.11.0. @nitins17 , @learning-to-play 
------------------------

### System information

-   **Have I written custom code (as opposed to using a stock example script
    provided in TensorFlow)**: no
-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 20.04 aarch64
-   **Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue
    happens on a mobile device**: no
-   **TensorFlow installed from (source or binary)**: source
-   **TensorFlow version (use command below)**: v2.11.0
-   **Python version**: 3.8.10
-   **Bazel version (if compiling from source)**: 5.3.0
-   **GCC/Compiler version (if compiling from source)**: gcc 9.4.0/9.3.0
-   **CUDA/cuDNN version**: 11.8/8.7
-   **GPU model and memory**: RTX A5000
-   **Exact command to reproduce**: follow https://www.tensorflow.org/install/source with cuda enabled
	- docker run -it -rm nvidia/cuda:11.8.0-cudnn8-devel-ubuntu20.04
	- wget https://github.com/bazelbuild/bazel/releases/download/5.3.0/bazel-5.3.0-linux-arm64
	- git clone -b v2.11.0 https://github.com/tensorflow/tensorflow.git
	- ./config with CUDA support? [y/N]: y
	- bazel build --config=cuda --enable_runfiles //tensorflow/tools/pip_package:build_pip_package 

### Describe the problem
TF is configured with gcc and compiles until linking, looks for clang (10.0.0 installed but not used) and fails. Same issue if downgrading to gcc 9.3. Worked on older v2.8 or v2.4. Works on x86_64.

### Source code / logs
ERROR: /home/ubuntu/tensorflow/tensorflow/tensorflow/BUILD:1263:21: Linking tensorflow/libtensorflow_cc.so.2.13.0 failed: (Exit 1): **crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc** @bazel-out/aarch64-opt/bin/tensorflow/libtensorflow_cc.so.2.13.0-2.params
collect2: error: ld returned 1 exit status
...
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 3256.167s, Critical Path: 1408.99s
INFO: 37283 processes: 12379 internal, 24904 local.
FAILED: Build did NOT complete successfully