Allow selecting enabled targets
Hi, @SuryanarayanaY 

Could you please look into this issue? Thank you! Hi @RoboTux ,

Could you please confirm the sequence of commands you used for build.
Also please provide the error logs that may help us to analyse the issue. Thanks! 1. Delete bazel cache
2. Run `bazel build --config=dbg -c opt tensorflow/compiler/mlir:tf-opt`
3. ls bazel-out/*-opt/bin/external/llvm-project/llvm/lib/Target

Which for me shows AArch64, AMDGPU, ARM and X86. I have a WIP patch that allow me to select the targets I want enabled and have for instance only AArch64. This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Thank you.
 @RoboTux ,

Thanks for confirmation.If this can be doable then definitely it will save some time AFAIK. Do you have plan to raise a PR to enable this feature? Our team may review and if all well may approve. Thanks! > @RoboTux ,
> 
> Thanks for confirmation.If this can be doable then definitely it will save some time AFAIK. Do you have plan to raise a PR to enable this feature? Our team may review and if all well may approve. Thanks!

Absolutely, I was waiting on someone to confirm it's a desirable feature. I'll be running some tests and publish it shortly. @RoboTux ,

I have tried to build from source on Ubuntu 20 VM and I got the Targets as `AMDGPU  X86`. Please refer the attached log.

```
(bazel) suryanarayanay@surya-ubuntu20:~/tensorflow$ bazel build --config=dbg -c opt tensorflow/compiler/mlir:tf-opt
Starting local Bazel server and connecting to it...
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=100
INFO: Reading rc options for 'build' from /home/suryanarayanay/tensorflow/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /home/suryanarayanay/tensorflow/.bazelrc:
  'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility --deleted_packages=tensorflow/compiler/mlir/tfrt,tensorflow/compiler/mlir/tfrt/benchmarks,tensorflow/compiler/mlir/tfrt/jit/python_binding,tensorflow/compiler/mlir/tfrt/jit/transforms,tensorflow/compiler/mlir/tfrt/python_tests,tensorflow/compiler/mlir/tfrt/tests,tensorflow/compiler/mlir/tfrt/tests/ir,tensorflow/compiler/mlir/tfrt/tests/analysis,tensorflow/compiler/mlir/tfrt/tests/jit,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_tfrt,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_jitrt,tensorflow/compiler/mlir/tfrt/tests/tf_to_corert,tensorflow/compiler/mlir/tfrt/tests/tf_to_tfrt_data,tensorflow/compiler/mlir/tfrt/tests/saved_model,tensorflow/compiler/mlir/tfrt/transforms/lhlo_gpu_to_tfrt_gpu,tensorflow/core/runtime_fallback,tensorflow/core/runtime_fallback/conversion,tensorflow/core/runtime_fallback/kernel,tensorflow/core/runtime_fallback/opdefs,tensorflow/core/runtime_fallback/runtime,tensorflow/core/runtime_fallback/util,tensorflow/core/tfrt/eager,tensorflow/core/tfrt/eager/backends/cpu,tensorflow/core/tfrt/eager/backends/gpu,tensorflow/core/tfrt/eager/core_runtime,tensorflow/core/tfrt/eager/cpp_tests/core_runtime,tensorflow/core/tfrt/gpu,tensorflow/core/tfrt/run_handler_thread_pool,tensorflow/core/tfrt/runtime,tensorflow/core/tfrt/saved_model,tensorflow/core/tfrt/graph_executor,tensorflow/core/tfrt/saved_model/tests,tensorflow/core/tfrt/tpu,tensorflow/core/tfrt/utils
INFO: Found applicable config definition build:short_logs in file /home/suryanarayanay/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /home/suryanarayanay/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:dbg in file /home/suryanarayanay/tensorflow/.bazelrc: -c dbg --per_file_copt=+.*,-tensorflow.*@-g0 --per_file_copt=+tensorflow/core/kernels.*@-g0 --cxxopt -DTF_LITE_DISABLE_X86_NEON --copt -DDEBUG_BUILD
INFO: Found applicable config definition build:linux in file /home/suryanarayanay/tensorflow/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --copt=-Wno-error=unused-but-set-variable --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --distinct_host_configuration=false --experimental_guard_against_concurrent_changes
INFO: Found applicable config definition build:dynamic_kernels in file /home/suryanarayanay/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
INFO: Analyzed target //tensorflow/compiler/mlir:tf-opt (372 packages loaded, 24423 targets configured).
INFO: Found 1 target...
Target //tensorflow/compiler/mlir:tf-opt up-to-date:
  bazel-bin/tensorflow/compiler/mlir/tf-opt
INFO: Elapsed time: 4232.262s, Critical Path: 377.67s
INFO: 11031 processes: 1009 internal, 10022 local.
INFO: Build completed successfully, 11031 total actions
(bazel) suryanarayanay@surya-ubuntu20:~/tensorflow$ ls bazel-out/*-opt/bin/external/llvm-project/llvm/lib/Target
AMDGPU  X86
```
I am not getting AArch64 ARM for targets. Could you please cross check and confirm ? Please note I have tested on Ubuntu 20 machine.

Thanks!
 > @RoboTux ,
> 
> I have tried to build from source on Ubuntu 20 VM and I got the Targets as `AMDGPU X86`. Please refer the attached log.
> 
> ```
> (bazel) suryanarayanay@surya-ubuntu20:~/tensorflow$ bazel build --config=dbg -c opt tensorflow/compiler/mlir:tf-opt
> Starting local Bazel server and connecting to it...
> INFO: Options provided by the client:
>   Inherited 'common' options: --isatty=1 --terminal_columns=100
> INFO: Reading rc options for 'build' from /home/suryanarayanay/tensorflow/.bazelrc:
>   Inherited 'common' options: --experimental_repo_remote_exec
> INFO: Reading rc options for 'build' from /home/suryanarayanay/tensorflow/.bazelrc:
>   'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility --deleted_packages=tensorflow/compiler/mlir/tfrt,tensorflow/compiler/mlir/tfrt/benchmarks,tensorflow/compiler/mlir/tfrt/jit/python_binding,tensorflow/compiler/mlir/tfrt/jit/transforms,tensorflow/compiler/mlir/tfrt/python_tests,tensorflow/compiler/mlir/tfrt/tests,tensorflow/compiler/mlir/tfrt/tests/ir,tensorflow/compiler/mlir/tfrt/tests/analysis,tensorflow/compiler/mlir/tfrt/tests/jit,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_tfrt,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_jitrt,tensorflow/compiler/mlir/tfrt/tests/tf_to_corert,tensorflow/compiler/mlir/tfrt/tests/tf_to_tfrt_data,tensorflow/compiler/mlir/tfrt/tests/saved_model,tensorflow/compiler/mlir/tfrt/transforms/lhlo_gpu_to_tfrt_gpu,tensorflow/core/runtime_fallback,tensorflow/core/runtime_fallback/conversion,tensorflow/core/runtime_fallback/kernel,tensorflow/core/runtime_fallback/opdefs,tensorflow/core/runtime_fallback/runtime,tensorflow/core/runtime_fallback/util,tensorflow/core/tfrt/eager,tensorflow/core/tfrt/eager/backends/cpu,tensorflow/core/tfrt/eager/backends/gpu,tensorflow/core/tfrt/eager/core_runtime,tensorflow/core/tfrt/eager/cpp_tests/core_runtime,tensorflow/core/tfrt/gpu,tensorflow/core/tfrt/run_handler_thread_pool,tensorflow/core/tfrt/runtime,tensorflow/core/tfrt/saved_model,tensorflow/core/tfrt/graph_executor,tensorflow/core/tfrt/saved_model/tests,tensorflow/core/tfrt/tpu,tensorflow/core/tfrt/utils
> INFO: Found applicable config definition build:short_logs in file /home/suryanarayanay/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
> INFO: Found applicable config definition build:v2 in file /home/suryanarayanay/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
> INFO: Found applicable config definition build:dbg in file /home/suryanarayanay/tensorflow/.bazelrc: -c dbg --per_file_copt=+.*,-tensorflow.*@-g0 --per_file_copt=+tensorflow/core/kernels.*@-g0 --cxxopt -DTF_LITE_DISABLE_X86_NEON --copt -DDEBUG_BUILD
> INFO: Found applicable config definition build:linux in file /home/suryanarayanay/tensorflow/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --copt=-Wno-error=unused-but-set-variable --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --distinct_host_configuration=false --experimental_guard_against_concurrent_changes
> INFO: Found applicable config definition build:dynamic_kernels in file /home/suryanarayanay/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
> INFO: Analyzed target //tensorflow/compiler/mlir:tf-opt (372 packages loaded, 24423 targets configured).
> INFO: Found 1 target...
> Target //tensorflow/compiler/mlir:tf-opt up-to-date:
>   bazel-bin/tensorflow/compiler/mlir/tf-opt
> INFO: Elapsed time: 4232.262s, Critical Path: 377.67s
> INFO: 11031 processes: 1009 internal, 10022 local.
> INFO: Build completed successfully, 11031 total actions
> (bazel) suryanarayanay@surya-ubuntu20:~/tensorflow$ ls bazel-out/*-opt/bin/external/llvm-project/llvm/lib/Target
> AMDGPU  X86
> ```
> 
> I am not getting AArch64 ARM for targets. Could you please cross check and confirm ? Please note I have tested on Ubuntu 20 machine.
> 
> Thanks!

I can confirm when building on x86 Ubuntu 22.04. I see that utils/bazel/llvm-project-overlay/llvm/BUILD.bazel has some config_setting that check the cpu value which might explain why that is. But then it's strange to have x86 when doing an AArch64 build. I'll investigate further but that most likely means reworking the patch since with it by default it builds all CPU targets. Hi @RoboTux ,

I would like to know whether did you changes any code in .bazelrc or any other bazel files etc which might configured Target for different platforms.I have not changed any files and got build for only one Target. Please provide all the changes you have done(if any) to enable the build for multiple targets.

Thank you! Hi @SuryanarayanaY ,

Sorry for the slow reply. I've found why X86 when building on AArch64 but not other LLVM targets thanks to the great bazel query allpaths feature:

@llvm-project/llvm:X86CodeGen gets pulled from //tensorflow/compiler/xla/service/cpu:cpu_compiler, as well as lower down from //tensorflow/compile/xla/runtime:execution_engine and from //tensorflow/compiler/xla/mlir/runtime/transforms:jit_compiler.

Other targets dependency are guarded by select around targets in //tensorflow/tsl (e.g.arm_any or linux_ppc64le) or by if_llvm_<target>_available. Now the patch contributed in PR59995 does solve that but enables all CPU targets by default because I erroneously thought it was the default. I'll rework it to only enable the host CPU target by default.

> 
> I would like to know whether did you changes any code in .bazelrc or any other bazel files etc which might configured Target for different platforms.I have not changed any files and got build for only one Target. Please provide all the changes you have done(if any) to enable the build for multiple targets.
> 

To answer your question I have no .bazelrc except for the one in tensorflow and my .tf_configure.bazelrc is what comes by default when running configure:
```

build --action_env PYTHON_BIN_PATH="/usr/bin/python3.10"
build --action_env PYTHON_LIB_PATH="/usr/lib/python3/dist-packages"
build --python_path="/usr/bin/python3.10"
build:opt --copt=-Wno-sign-compare
build:opt --host_copt=-Wno-sign-compare
test --flaky_test_attempts=3
test --test_size_filters=small,medium
test:v1 --test_tag_filters=-benchmark-test,-no_oss,-oss_excluded,-gpu,-oss_serial
test:v1 --build_tag_filters=-benchmark-test,-no_oss,-oss_excluded,-gpu
test:v2 --test_tag_filters=-benchmark-test,-no_oss,-oss_excluded,-gpu,-oss_serial,-v1only
test:v2 --build_tag_filters=-benchmark-test,-no_oss,-oss_excluded,-gpu,-v1only
```

As mentioned, this is on AArch64 host which is what makes the difference. @RoboTux ,

The build I was done was on a machine with `X86` architecture which seems building the target for that particular architecture only. Whereas as per your investigation for `AArch64` it is building for both `AArch64` as well as `X86` which might be the reason for getting different Targets for me.

Please continue working on the Patch and I hope this will be approved by dev-team.

Thanks!





 Patch has been merged. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59595">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59595">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Build/Install

### Have you reproduced the bug with TF nightly?

Yes

### Source

source

### Tensorflow Version

master

### Custom Code

Yes

### OS Platform and Distribution

Linux Ubuntu 22.04

### Mobile device

_No response_

### Python version

3.10

### Bazel version

5.3.0

### GCC/Compiler version

11.3

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current Behaviour?

```shell
Many parts of TensorFlow (e.g. TensorFlow XLA execution_engine in tensorflow/compiler/xla/runtime) build support for several targets unconditionally. This leads to a lot of extra build time when that support depends on LLVM support and thus extra backend needing to be enabled in LLVM. I'd like to work on a patch to be able to select targets to be enabled at configure or build time, keeping the current default of enabling most targets. I would also like to enable AArch64 target by default for aot and kernel_gen since ARM target is already enabled unconditionaly at the moment.
```


### Standalone code to reproduce the issue

```shell
Following the steps in tensorflow/compiler/mlir/README.md "Using local LLVM repo" with only AArch64 in targets.bzl fails to build.
```


### Relevant log output

_No response_</details>