tensorflow 2.16.1 build error: Compiling xla/service/cpu/onednn_matmul.cc failed
@spamming4 Please ensure you have OneDNN installed on your system. Kindly refer to the official OneDNN installation guide for your operating system: https://onnxruntime.ai/docs/execution-providers/oneDNN-ExecutionProvider.html
also verify that the installed OneDNN version is compatible with TensorFlow 2.16.1. 
Thank you! > Please ensure you have OneDNN installed on your system. Kindly refer to the official OneDNN installation guide for your operating system: https://onnxruntime.ai/docs/execution-providers/oneDNN-ExecutionProvider.html

Did ChatGPT write this?

+@agramesh1 @spamming4 the TF build option --config=mkl is part of the Intel tensorFlow distribution and no longer supported as of  TF 2.15 (see https://github.com/Intel-tensorflow/tensorflow/releases/tag/v2.14.0)  as all the oneDNN optimization as currently available with the official build.  You should be able to compile TF without using --config=mkl .
 So if I understand correctly, we build without `--config=mkl` and install `intel-extension-for-tensorflow` alongside tf? @sirfz, oneDNN support is enabled by default on Windows in Tensorflow [v2.15](https://github.com/tensorflow/tensorflow/releases/tag/v2.15.0) and newer. You don't need to do anything special. Just grab Tensorflow build from whatever channel you like or build from source. > @spamming4 Please ensure you have OneDNN installed on your system. Kindly refer to the official OneDNN installation guide for your operating system: https://onnxruntime.ai/docs/execution-providers/oneDNN-ExecutionProvider.html also verify that the installed OneDNN version is compatible with TensorFlow 2.16.1. Thank you!

[sushreebarsa](https://github.com/sushreebarsa), thank you, that helped to resolve the issue. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/63360">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/63360">No</a>
 I ran into the same error and found the answers here misleading.

I can confirm that the prebuilt package (`pip install tensorflow==2.16.1`) gives competitive performance to anything I was able to build from source. Had to set `TF_ENABLE_ONEDNN_OPTS=1` for that.

But to clarify:
> the TF build option --config=mkl is part of the Intel tensorFlow distribution and no longer supported as of TF 2.15

I build the official TF 2.16.1 with and without `--config=mkl` and saw a clear difference in performance (even with `with_xla_support=false`which I had to set to get to compile). So this statement holds only for [intel-tensorflow](https://github.com/Intel-tensorflow/tensorflow)? Even in intel-tensorflow the option still seems to exist in .bazelrc?

> You should be able to compile TF without using --config=mkl .

True, that "solves" the original compilation error (which is about official TF 2.16.1) but results in worse performance.

> oneDNN support is enabled by default on Windows in Tensorflow [v2.15](https://github.com/tensorflow/tensorflow/releases/tag/v2.15.0) and newer

Who asked about windows? 😉 On Ubuntu I had to set `TF_ENABLE_ONEDNN_OPTS=1` as I said, not sure if that's the case for everyone? But yes, other than that works out of the box. 🙌 

> sushreebarsa, thank you, that helped to resolve the issue.

Really? 😅 Build instructions for ONNX? There are no necessary steps mentioned regarding oneDNN on https://www.tensorflow.org/install/source when using `--config==mkl`. Are there any?

---

In any case, the original error
```
error: cannot convert ‘std::unique_ptr<tsl::OneDnnThreadPool>::pointer’ {aka ‘tsl::OneDnnThreadPool*’} to ‘dnnl::threadpool_interop::threadpool_iface*’
```
is not solved. And the status is "won't fix"? I am not working on XLA:CPU. @penpornk maybe you can comment on the compile error reported here?
### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

2.16.1

### Custom code

No

### OS platform and distribution

Linux Ubuntu 22.04

### Mobile device

_No response_

### Python version

3.11.8

### Bazel version

6.5.0

### GCC/compiler version

11.4.0

### CUDA/cuDNN version

12.4/9.0.0.312

### GPU model and memory

NVIDIA GeForce 940MX

### Current behavior?

INFO: Reading 'startup' options from ~/Documents/dev/git/tensorflow/.bazelrc: --windows_enable_symlinks
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=211
INFO: Reading rc options for 'build' from ~/Documents/dev/git/tensorflow/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from ~/Documents/dev/git/tensorflow/.bazelrc:
  'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --features=-force_no_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility
INFO: Reading rc options for 'build' from ~/Documents/dev/git/tensorflow/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=~/Documents/dev/programs/miniconda3/envs/tf/bin/python3 --action_env PYTHON_LIB_PATH=~/Documents/dev/programs/miniconda3/envs/tf/lib/python3.11/site-packages --python_path=~/Documents/dev/programs/miniconda3/envs/tf/bin/python3 --action_env CUDA_TOOLKIT_PATH=/usr/local/cuda-12.3 --action_env TF_CUDA_COMPUTE_CAPABILITIES=5.0 --action_env LD_LIBRARY_PATH=/usr/lib/libreoffice/program:/usr/local/cuda/targets/x86_64-linux/lib:/usr/lib/x86_64-linux-gnu --action_env GCC_HOST_COMPILER_PATH=/usr/bin/x86_64-linux-gnu-gcc-11 --config=cuda
INFO: Found applicable config definition build:short_logs in file ~/Documents/dev/git/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file ~/Documents/dev/git/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:cuda in file ~/Documents/dev/git/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
INFO: Found applicable config definition build:mkl in file ~/Documents/dev/git/tensorflow/.bazelrc: --define=build_with_mkl=true --define=enable_mkl=true --define=tensorflow_mkldnn_contraction_kernel=0 --define=build_with_openmp=true -c opt
INFO: Found applicable config definition build:opt in file ~/Documents/dev/git/tensorflow/.tf_configure.bazelrc: --copt=-Wno-sign-compare --host_copt=-Wno-sign-compare
INFO: Found applicable config definition build:linux in file ~/Documents/dev/git/tensorflow/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --copt=-Wno-error=unused-but-set-variable --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --experimental_guard_against_concurrent_changes
INFO: Found applicable config definition build:dynamic_kernels in file ~/Documents/dev/git/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
INFO: Analyzed target //tensorflow/tools/pip_package:build_pip_package (711 packages loaded, 51601 targets configured).
INFO: Found 1 target...
ERROR: ~/.cache/bazel/_bazel_vyepishov/cf67b2b2e967476eb2b1ee98e33ab5bd/external/local_xla/xla/service/cpu/BUILD:1638:11: Compiling xla/service/cpu/onednn_matmul.cc failed: (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command (from target @local_xla//xla/service/cpu:onednn_matmul) external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF bazel-out/k8-opt/bin/external/local_xla/xla/service/cpu/_objs/onednn_matmul/onednn_matmul.pic.d ... (remaining 229 arguments skipped)
In file included from external/local_xla/xla/shape.h:28,
                 from external/local_xla/xla/service/cpu/onednn_matmul.h:21,
                 from external/local_xla/xla/service/cpu/onednn_matmul.cc:18:
external/local_xla/xla/layout.h:377:18: warning: ‘xla::Layout::DimInfo::dim_level_type’ is too small to hold all values of ‘enum xla::DimLevelType’
  377 |     DimLevelType dim_level_type : 6;
      |                  ^~~~~~~~~~~~~~
external/local_xla/xla/layout.h:389:17: warning: ‘xla::Layout::index_primitive_type_’ is too small to hold all values of ‘enum xla::PrimitiveType’
  389 |   PrimitiveType index_primitive_type_ : 8;
      |                 ^~~~~~~~~~~~~~~~~~~~~
external/local_xla/xla/layout.h:390:17: warning: ‘xla::Layout::pointer_primitive_type_’ is too small to hold all values of ‘enum xla::PrimitiveType’
  390 |   PrimitiveType pointer_primitive_type_ : 8;
      |                 ^~~~~~~~~~~~~~~~~~~~~~~
external/local_xla/xla/service/cpu/onednn_matmul.cc: In function ‘void xla::cpu::__xla_cpu_runtime_OneDnnMatMul(void*, void**)’:
external/local_xla/xla/service/cpu/onednn_matmul.cc:186:68: error: cannot convert ‘std::unique_ptr<tsl::OneDnnThreadPool>::pointer’ {aka ‘tsl::OneDnnThreadPool*’} to ‘dnnl::threadpool_interop::threadpool_iface*’
  186 |   auto onednn_stream = MakeOneDnnStream(cpu_engine, thread_pool.get());
      |                                                     ~~~~~~~~~~~~~~~^~
      |                                                                    |
      |                                                                    std::unique_ptr<tsl::OneDnnThreadPool>::pointer {aka tsl::OneDnnThreadPool*}
external/local_xla/xla/service/cpu/onednn_matmul.cc:148:49: note:   initializing argument 2 of ‘dnnl::stream xla::cpu::{anonymous}::MakeOneDnnStream(const dnnl::engine&, dnnl::threadpool_interop::threadpool_iface*)’
  148 |     dnnl::threadpool_interop::threadpool_iface* thread_pool) {
      |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~
external/local_xla/xla/service/cpu/onednn_matmul.cc: In function ‘void xla::cpu::__xla_cpu_runtime_OneDnnMatMulReorder(void*, void**)’:
external/local_xla/xla/service/cpu/onednn_matmul.cc:322:68: error: cannot convert ‘std::unique_ptr<tsl::OneDnnThreadPool>::pointer’ {aka ‘tsl::OneDnnThreadPool*’} to ‘dnnl::threadpool_interop::threadpool_iface*’
  322 |   auto onednn_stream = MakeOneDnnStream(cpu_engine, thread_pool.get());
      |                                                     ~~~~~~~~~~~~~~~^~
      |                                                                    |
      |                                                                    std::unique_ptr<tsl::OneDnnThreadPool>::pointer {aka tsl::OneDnnThreadPool*}
external/local_xla/xla/service/cpu/onednn_matmul.cc:148:49: note:   initializing argument 2 of ‘dnnl::stream xla::cpu::{anonymous}::MakeOneDnnStream(const dnnl::engine&, dnnl::threadpool_interop::threadpool_iface*)’
  148 |     dnnl::threadpool_interop::threadpool_iface* thread_pool) {
      |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 16142.186s, Critical Path: 328.40s
INFO: 25824 processes: 8831 internal, 16993 local.
FAILED: Build did NOT complete successfully

### Standalone code to reproduce the issue

```shell
bazel build --config=mkl --config=opt //tensorflow/tools/pip_package:build_pip_package
```


### Relevant log output

```shell
INFO: Reading 'startup' options from ~/Documents/dev/git/tensorflow/.bazelrc: --windows_enable_symlinks
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=211
INFO: Reading rc options for 'build' from ~/Documents/dev/git/tensorflow/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from ~/Documents/dev/git/tensorflow/.bazelrc:
  'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --features=-force_no_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility
INFO: Reading rc options for 'build' from ~/Documents/dev/git/tensorflow/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=~/Documents/dev/programs/miniconda3/envs/tf/bin/python3 --action_env PYTHON_LIB_PATH=~/Documents/dev/programs/miniconda3/envs/tf/lib/python3.11/site-packages --python_path=~/Documents/dev/programs/miniconda3/envs/tf/bin/python3 --action_env CUDA_TOOLKIT_PATH=/usr/local/cuda-12.3 --action_env TF_CUDA_COMPUTE_CAPABILITIES=5.0 --action_env LD_LIBRARY_PATH=/usr/lib/libreoffice/program:/usr/local/cuda/targets/x86_64-linux/lib:/usr/lib/x86_64-linux-gnu --action_env GCC_HOST_COMPILER_PATH=/usr/bin/x86_64-linux-gnu-gcc-11 --config=cuda
INFO: Found applicable config definition build:short_logs in file ~/Documents/dev/git/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file ~/Documents/dev/git/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:cuda in file ~/Documents/dev/git/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
INFO: Found applicable config definition build:mkl in file ~/Documents/dev/git/tensorflow/.bazelrc: --define=build_with_mkl=true --define=enable_mkl=true --define=tensorflow_mkldnn_contraction_kernel=0 --define=build_with_openmp=true -c opt
INFO: Found applicable config definition build:opt in file ~/Documents/dev/git/tensorflow/.tf_configure.bazelrc: --copt=-Wno-sign-compare --host_copt=-Wno-sign-compare
INFO: Found applicable config definition build:linux in file ~/Documents/dev/git/tensorflow/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --copt=-Wno-error=unused-but-set-variable --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --experimental_guard_against_concurrent_changes
INFO: Found applicable config definition build:dynamic_kernels in file ~/Documents/dev/git/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
INFO: Analyzed target //tensorflow/tools/pip_package:build_pip_package (711 packages loaded, 51601 targets configured).
INFO: Found 1 target...
ERROR: ~/.cache/bazel/_bazel_vyepishov/cf67b2b2e967476eb2b1ee98e33ab5bd/external/local_xla/xla/service/cpu/BUILD:1638:11: Compiling xla/service/cpu/onednn_matmul.cc failed: (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command (from target @local_xla//xla/service/cpu:onednn_matmul) external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF bazel-out/k8-opt/bin/external/local_xla/xla/service/cpu/_objs/onednn_matmul/onednn_matmul.pic.d ... (remaining 229 arguments skipped)
In file included from external/local_xla/xla/shape.h:28,
                 from external/local_xla/xla/service/cpu/onednn_matmul.h:21,
                 from external/local_xla/xla/service/cpu/onednn_matmul.cc:18:
external/local_xla/xla/layout.h:377:18: warning: ‘xla::Layout::DimInfo::dim_level_type’ is too small to hold all values of ‘enum xla::DimLevelType’
  377 |     DimLevelType dim_level_type : 6;
      |                  ^~~~~~~~~~~~~~
external/local_xla/xla/layout.h:389:17: warning: ‘xla::Layout::index_primitive_type_’ is too small to hold all values of ‘enum xla::PrimitiveType’
  389 |   PrimitiveType index_primitive_type_ : 8;
      |                 ^~~~~~~~~~~~~~~~~~~~~
external/local_xla/xla/layout.h:390:17: warning: ‘xla::Layout::pointer_primitive_type_’ is too small to hold all values of ‘enum xla::PrimitiveType’
  390 |   PrimitiveType pointer_primitive_type_ : 8;
      |                 ^~~~~~~~~~~~~~~~~~~~~~~
external/local_xla/xla/service/cpu/onednn_matmul.cc: In function ‘void xla::cpu::__xla_cpu_runtime_OneDnnMatMul(void*, void**)’:
external/local_xla/xla/service/cpu/onednn_matmul.cc:186:68: error: cannot convert ‘std::unique_ptr<tsl::OneDnnThreadPool>::pointer’ {aka ‘tsl::OneDnnThreadPool*’} to ‘dnnl::threadpool_interop::threadpool_iface*’
  186 |   auto onednn_stream = MakeOneDnnStream(cpu_engine, thread_pool.get());
      |                                                     ~~~~~~~~~~~~~~~^~
      |                                                                    |
      |                                                                    std::unique_ptr<tsl::OneDnnThreadPool>::pointer {aka tsl::OneDnnThreadPool*}
external/local_xla/xla/service/cpu/onednn_matmul.cc:148:49: note:   initializing argument 2 of ‘dnnl::stream xla::cpu::{anonymous}::MakeOneDnnStream(const dnnl::engine&, dnnl::threadpool_interop::threadpool_iface*)’
  148 |     dnnl::threadpool_interop::threadpool_iface* thread_pool) {
      |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~
external/local_xla/xla/service/cpu/onednn_matmul.cc: In function ‘void xla::cpu::__xla_cpu_runtime_OneDnnMatMulReorder(void*, void**)’:
external/local_xla/xla/service/cpu/onednn_matmul.cc:322:68: error: cannot convert ‘std::unique_ptr<tsl::OneDnnThreadPool>::pointer’ {aka ‘tsl::OneDnnThreadPool*’} to ‘dnnl::threadpool_interop::threadpool_iface*’
  322 |   auto onednn_stream = MakeOneDnnStream(cpu_engine, thread_pool.get());
      |                                                     ~~~~~~~~~~~~~~~^~
      |                                                                    |
      |                                                                    std::unique_ptr<tsl::OneDnnThreadPool>::pointer {aka tsl::OneDnnThreadPool*}
external/local_xla/xla/service/cpu/onednn_matmul.cc:148:49: note:   initializing argument 2 of ‘dnnl::stream xla::cpu::{anonymous}::MakeOneDnnStream(const dnnl::engine&, dnnl::threadpool_interop::threadpool_iface*)’
  148 |     dnnl::threadpool_interop::threadpool_iface* thread_pool) {
      |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 16142.186s, Critical Path: 328.40s
INFO: 25824 processes: 8831 internal, 16993 local.
FAILED: Build did NOT complete successfully
```
