[MKL] Manylinux wheel issue on clang
Hi @ke1ding,

Tensorflow officially supports build with GCC as per tested configurations. Build with Clang is not documented properly and hence I don't have much information. Recommending to Concern Team.

@TensorFlow-MKL , Could you please have a look into the issue.

Thanks! Hi @ke1ding,

Did you build the package inside a manylinux2014 docker container? According to the [manylinux2014 policy](https://peps.python.org/pep-0599/#the-manylinux2014-policy), the maximum glibc version supported is 2.17 which might be the issue.

Can you also run ldd --version and share the output?

Thanks! @ke1ding 
I think this issue is about manylinux usage issue instead of TensorFlow building with MKL.

1. 
According to the manylinux repo readme: https://github.com/pypa/manylinux

```
|manylinux2014 | pip >= 19.3 | 3.7.8+, 3.8.4+, 3.9.0+|

...

manylinux2014 (CentOS 7 based)

Toolchain: GCC 10

    x86_64 image: quay.io/pypa/manylinux2014_x86_64
    i686 image: quay.io/pypa/manylinux2014_i686
    aarch64 image: quay.io/pypa/manylinux2014_aarch64
    ppc64le image: quay.io/pypa/manylinux2014_ppc64le
    s390x image: quay.io/pypa/manylinux2014_s390x

```
If you want to build with manylinux2014, please following the requirement of manylinux2014 as above.

2.
According to the error log, there is a long dependence chain in your case.
You could fix the last un-compliant issue (so file) firstly.

Thank you!




 Hi @ke1ding, @yehudaorel ,

I agree to the fact that GCC version capped at 2.17 for Manylinux_2014 as mentioned in above [comment](https://github.com/tensorflow/tensorflow/issues/60608#issuecomment-1552294597) also. But here the built is done using Clang but not GCC and AFAIK, Clang builds may not need GCC.

I am not much aware of the `auditwheel` package that is being used here for making the built binaries to manylinux_2014 compatible.May be the problem is with `manylinux_2_17(aka manylinux2014)` which indicates max supported GCC version is 2.17v. But the question is the shared objects generated by Clang build still depends upon GCC ? I am not much into compiler design things and liked to hear any pointers here.

Thanks!

 @SuryanarayanaY @ke1ding Building with a recent compiler while maintaining manylinux2014 compatibility is a very complex task. Since https://github.com/tensorflow/tensorflow/commit/ba2f7926467cd0b2b7dfea899c6f507b719e7669 the CI builds for Linux from TensorFlow are built using clang-16. You can use the SIG Build docker containers and the appropriate TensorFlow toolchain if you want to build your own or just study what needs to be done. Hi @ke1ding , 

This issue may be caused by compiling against a newer version of glibc than manylinux_2014 supports. manylinux_2014 must be compatible with glibc 2.17, but glibc_2.29 is being used. 

You may refer to this [script](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/tf_sig_build_dockerfiles/builder.devtoolset/build_devtoolset.sh) and this [Dockerfile](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/tf_sig_build_dockerfiles/Dockerfile#L6-L16) to see how to install glibc 2.17 in your sysroot and build the toolchain with manylinux.  @ke1ding ,

Could you please refer to above comments and confirm whether glibc can be downgraded to 2.17 and tried the build ?

Please confirm the status. Thanks!
 @SuryanarayanaY Yes Thank you I will try. This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. Hi @kanglant @SuryanarayanaY 

 I referred the build_devtools.sh to build a docker image [build_devtoolset.sh](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/tf_sig_build_dockerfiles/builder.devtoolset/build_devtoolset.sh) for gcc, glibc and libstdc++, after that I install llvm/clang-16. Before I build tensorflow, I checked the glibc version in the docker image by 
`ldd --version && getconf GNU_LIBC_VERSION`.
It displays  as: 
`ldd (Ubuntu GLIBC 2.31-0ubuntu9.9) 2.31
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. 
glibc 2.31`

For using clang-16, do we need more steps to downgrade the GLIBC version again?

P.S. I will try to generate a docker image by tf_sig_build_dockerfiles [Dockerfile](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/tf_sig_build_dockerfiles/Dockerfile#L6-L16) @yuntaolu The downgraded glibc is installed into a sysroot under /dt9 so this will not show up in your check as it has not replaced the system glibc. To build with clang and the 2.17 version of glibc you need to pass
```
--crosstool_top="@sigbuild-r2.14-clang_config_cuda//crosstool:toolchain"
```
on the TensorFlow build command line. > @yuntaolu The downgraded glibc is installed into a sysroot under /dt9 so this will not show up in your check as it has not replaced the system glibc. To build with clang and the 2.17 version of glibc you need to pass
> 
> ```
> --crosstool_top="@sigbuild-r2.14-clang_config_cuda//crosstool:toolchain"
> ```
> 
> on the TensorFlow build command line.

I applied the suggestion from @elfringham, and added the parameter ""--crosstool_top=\"@sigbuild-r2.13-clang_config_cuda//crosstool:toolchain\"" . Finally, I built TensorFlow manylinux2014 wheels successfully. 

Thanks for all your help with the manylinux2014 build with clang. 
 Thank you all! As problem solved I'll close this issue. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60608">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60608">No</a>
 @aice-support
<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

Yes

### Source

source

### Tensorflow Version

tf2.13rc0

### Custom Code

No

### OS Platform and Distribution

Ubuntu 20.04

### Mobile device

_No response_

### Python version

3.8

### Bazel version

5.3

### GCC/Compiler version

Clang 16

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current Behaviour?

When building with clang 16, config=mkl, we can successfully built. But when we tried to use auditwheel package to make it manylinux_2014 compatible, we faced error like:
```
_solib_k8/_U_S_Stensorflow_Stsl_Smkl_Cmkl_Ulibs_Ulinux___Uexternal_Sllvm_Uopenmp/libiomp5.so is manylinux_2_17(aka manylinux2014) compliant.
tensorflow/libtensorflow_cc.so.2 is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_framework.so.2
libiomp5.so
libc.so.6	offending versions: GLIBC_2.28, GLIBC_2.27
libm.so.6	offending versions: GLIBC_2.23, GLIBC_2.29, GLIBC_2.27
tensorflow/libtensorflow_framework.so.2 is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libm.so.6	offending versions: GLIBC_2.29, GLIBC_2.27
libiomp5.so
libc.so.6	offending versions: GLIBC_2.18
tensorflow/compiler/tf2tensorrt/_pywrap_py_utils.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libiomp5.so
libtensorflow_framework.so.2
tensorflow/compiler/tf2xla/ops/_xla_ops.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_framework.so.2
tensorflow/core/kernels/libtfkernel_sobol_op.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_framework.so.2
libm.so.6	offending versions: GLIBC_2.29
tensorflow/include/external/llvm_openmp/libiomp5.so is manylinux_2_17(aka manylinux2014) compliant.
tensorflow/lite/experimental/microfrontend/python/ops/_audio_microfrontend_op.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libm.so.6	offending versions: GLIBC_2.27
libtensorflow_framework.so.2
tensorflow/lite/python/analyzer_wrapper/_pywrap_analyzer_wrapper.so is manylinux_2_17(aka manylinux2014) compliant.
tensorflow/lite/python/interpreter_wrapper/_pywrap_tensorflow_interpreter_wrapper.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_framework.so.2
libm.so.6	offending versions: GLIBC_2.29, GLIBC_2.27
tensorflow/lite/python/metrics/_pywrap_tensorflow_lite_metrics_wrapper.so is manylinux_2_17(aka manylinux2014) compliant.
tensorflow/lite/python/optimize/_pywrap_tensorflow_lite_calibration_wrapper.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_framework.so.2
libm.so.6	offending versions: GLIBC_2.29, GLIBC_2.27
tensorflow/python/_pywrap_dtensor_device.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_framework.so.2
libiomp5.so
_pywrap_tensorflow_internal.so
tensorflow/python/_pywrap_mlir.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libiomp5.so
libtensorflow_framework.so.2
_pywrap_tensorflow_internal.so
tensorflow/python/_pywrap_parallel_device.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libiomp5.so
libtensorflow_framework.so.2
_pywrap_tensorflow_internal.so
tensorflow/python/_pywrap_py_exception_registry.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_cc.so.2
_pywrap_tensorflow_internal.so
libtensorflow_framework.so.2
libiomp5.so
tensorflow/python/_pywrap_quantize_training.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
_pywrap_tensorflow_internal.so
libtensorflow_framework.so.2
libiomp5.so
tensorflow/python/_pywrap_sanitizers.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_framework.so.2
_pywrap_tensorflow_internal.so
libiomp5.so
tensorflow/python/_pywrap_tensorflow_internal.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_cc.so.2
libbfloat16.so.so
libiomp5.so
libtensorflow_framework.so.2
tensorflow/python/_pywrap_tfcompile.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
_pywrap_tensorflow_internal.so
libiomp5.so
libtensorflow_framework.so.2
libtensorflow_cc.so.2
tensorflow/python/_pywrap_tfe.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
_pywrap_tensorflow_internal.so
libtensorflow_cc.so.2
libtensorflow_framework.so.2
libiomp5.so
tensorflow/python/_pywrap_toco_api.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_framework.so.2
_pywrap_tensorflow_internal.so
libiomp5.so
tensorflow/python/flags_pybind.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
_pywrap_tensorflow_internal.so
libtensorflow_framework.so.2
libiomp5.so
tensorflow/python/autograph/impl/testing/pybind_for_testing.so is manylinux_2_17(aka manylinux2014) compliant.
tensorflow/python/client/_pywrap_debug_events_writer.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
_pywrap_tensorflow_internal.so
libtensorflow_framework.so.2
libiomp5.so
tensorflow/python/client/_pywrap_device_lib.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libiomp5.so
libtensorflow_framework.so.2
_pywrap_tensorflow_internal.so
tensorflow/python/client/_pywrap_events_writer.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_framework.so.2
libiomp5.so
_pywrap_tensorflow_internal.so
tensorflow/python/client/_pywrap_tf_session.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_framework.so.2
libiomp5.so
_pywrap_tensorflow_internal.so
tensorflow/python/data/experimental/service/_pywrap_server_lib.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libiomp5.so
libtensorflow_framework.so.2
_pywrap_tensorflow_internal.so
tensorflow/python/data/experimental/service/_pywrap_utils.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libiomp5.so
libtensorflow_framework.so.2
libm.so.6	offending versions: GLIBC_2.29
_pywrap_tensorflow_internal.so
tensorflow/python/framework/_dtypes.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_framework.so.2
libiomp5.so
_pywrap_tensorflow_internal.so
tensorflow/python/framework/_errors_test_helper.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
_pywrap_tensorflow_internal.so
libm.so.6	offending versions: GLIBC_2.29
libtensorflow_framework.so.2
libiomp5.so
tensorflow/python/framework/_op_def_library_pybind.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
_pywrap_tensorflow_internal.so
libiomp5.so
libtensorflow_framework.so.2
tensorflow/python/framework/_op_def_registry.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_framework.so.2
_pywrap_tensorflow_internal.so
libiomp5.so
tensorflow/python/framework/_op_def_util.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
_pywrap_tensorflow_internal.so
libtensorflow_framework.so.2
libiomp5.so
tensorflow/python/framework/_proto_comparators.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libiomp5.so
libtensorflow_framework.so.2
_pywrap_tensorflow_internal.so
tensorflow/python/framework/_python_memory_checker_helper.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_framework.so.2
_pywrap_tensorflow_internal.so
libiomp5.so
tensorflow/python/framework/_pywrap_python_api_dispatcher.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_framework.so.2
libiomp5.so
_pywrap_tensorflow_internal.so
tensorflow/python/framework/_pywrap_python_api_info.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_framework.so.2
libiomp5.so
_pywrap_tensorflow_internal.so
tensorflow/python/framework/_pywrap_python_api_parameter_converter.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_framework.so.2
_pywrap_tensorflow_internal.so
libiomp5.so
tensorflow/python/framework/_pywrap_python_op_gen.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_framework.so.2
_pywrap_tensorflow_internal.so
libiomp5.so
tensorflow/python/framework/_test_metrics_util.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_framework.so.2
_pywrap_tensorflow_internal.so
libiomp5.so
tensorflow/python/framework/fast_tensor_util.so is manylinux_2_17(aka manylinux2014) compliant.
tensorflow/python/grappler/_pywrap_tf_cluster.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
_pywrap_tensorflow_internal.so
libtensorflow_framework.so.2
libiomp5.so
tensorflow/python/grappler/_pywrap_tf_item.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libiomp5.so
libtensorflow_framework.so.2
_pywrap_tensorflow_internal.so
tensorflow/python/grappler/_pywrap_tf_optimizer.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_framework.so.2
_pywrap_tensorflow_internal.so
libiomp5.so
tensorflow/python/lib/core/_pywrap_custom_casts.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libiomp5.so
libtensorflow_framework.so.2
_pywrap_tensorflow_internal.so
tensorflow/python/lib/core/_pywrap_float8.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libiomp5.so
libtensorflow_framework.so.2
_pywrap_tensorflow_internal.so
tensorflow/python/lib/core/_pywrap_py_func.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_framework.so.2
libiomp5.so
_pywrap_tensorflow_internal.so
tensorflow/python/lib/io/_pywrap_file_io.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libiomp5.so
_pywrap_tensorflow_internal.so
libtensorflow_framework.so.2
tensorflow/python/lib/io/_pywrap_record_io.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_framework.so.2
libiomp5.so
_pywrap_tensorflow_internal.so
tensorflow/python/platform/_pywrap_cpu_feature_guard.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_framework.so.2
_pywrap_tensorflow_internal.so
libiomp5.so
tensorflow/python/platform/_pywrap_stacktrace_handler.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libiomp5.so
_pywrap_tensorflow_internal.so
libtensorflow_framework.so.2
tensorflow/python/platform/_pywrap_tf2.so is manylinux_2_17(aka manylinux2014) compliant.
tensorflow/python/profiler/internal/_pywrap_profiler.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
_pywrap_tensorflow_internal.so
libtensorflow_framework.so.2
libiomp5.so
tensorflow/python/profiler/internal/_pywrap_traceme.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libiomp5.so
libtensorflow_framework.so.2
_pywrap_tensorflow_internal.so
tensorflow/python/saved_model/pywrap_saved_model.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libiomp5.so
_pywrap_tensorflow_internal.so
libtensorflow_framework.so.2
tensorflow/python/util/_pywrap_checkpoint_reader.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libiomp5.so
libtensorflow_framework.so.2
_pywrap_tensorflow_internal.so
tensorflow/python/util/_pywrap_determinism.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libiomp5.so
libtensorflow_framework.so.2
_pywrap_tensorflow_internal.so
tensorflow/python/util/_pywrap_kernel_registry.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libiomp5.so
_pywrap_tensorflow_internal.so
libtensorflow_framework.so.2
tensorflow/python/util/_pywrap_nest.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
_pywrap_tensorflow_internal.so
libtensorflow_framework.so.2
libiomp5.so
tensorflow/python/util/_pywrap_stat_summarizer.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
_pywrap_tensorflow_internal.so
libiomp5.so
libtensorflow_framework.so.2
tensorflow/python/util/_pywrap_tensor_float_32_execution.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libiomp5.so
libtensorflow_framework.so.2
_pywrap_tensorflow_internal.so
tensorflow/python/util/_pywrap_tfprof.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_framework.so.2
libiomp5.so
_pywrap_tensorflow_internal.so
tensorflow/python/util/_pywrap_transform_graph.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libiomp5.so
libtensorflow_framework.so.2
_pywrap_tensorflow_internal.so
tensorflow/python/util/_pywrap_util_port.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libiomp5.so
libtensorflow_framework.so.2
_pywrap_tensorflow_internal.so
tensorflow/python/util/_pywrap_utils.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libiomp5.so
_pywrap_tensorflow_internal.so
libtensorflow_framework.so.2
tensorflow/python/util/_tf_stack.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libiomp5.so
libtensorflow_framework.so.2
_pywrap_tensorflow_internal.so
tensorflow/python/util/fast_module_type.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libtensorflow_framework.so.2
_pywrap_tensorflow_internal.so
libiomp5.so
tensorflow/tsl/python/lib/core/libbfloat16.so.so is manylinux_2_17(aka manylinux2014) compliant.
tensorflow/tsl/python/lib/core/pywrap_bfloat16.so is not manylinux_2_17(aka manylinux2014) compliant because it links the following forbidden libraries:
libbfloat16.so.so
```


### Standalone code to reproduce the issue

```shell
Bazel build option:

build --copt=-O3 --features=-layering_check --copt=-Wno-gnu-offsetof-extensions --copt=-Wformat --copt=-Wformat-security --copt=-fstack-protector --copt=-fPIC --copt=-fpic --linkopt=-Wl,-z,noexecstack --linkopt=-Wl,-z,relro --linkopt=-Wl,-z,now --linkopt=-fstack-protector --config=mkl --copt=-march=sandybridge
```


### Relevant log output

_No response_</details>