Tensorflow debug build reports the error: relocation R_X86_64_PC32 against undefined symbol `_ZN15stream_executor3gpu12_GLOBAL__N_120CUDABlasLtMatmulPlan14kMaxBatchCountE' can not be used when making a shared object
Hi @SuryanarayanaY I think the  error message you're encountering indicates an issue with linking a shared object in the build process of TensorFlow. Specifically, it mentions a problem with the libcublas_plugin.lo object file and the relocation of symbols. The error message also suggests recompiling with -fPIC to resolve the issue.

Here are some steps you can take to troubleshoot and potentially fix this error:

Recompile with -fPIC: You can add the -fPIC flag to the build process to ensure that the object files are compiled with Position Independent Code (PIC). This is often required for shared libraries. You might need to modify the build configuration to include this flag. Check the build script or configuration files to make this change.

Check CUDA Toolkit Compatibility: Ensure that the CUDA Toolkit version you have installed is compatible with the version of TensorFlow you are trying to build. Different versions of TensorFlow may require specific CUDA Toolkit versions. Make sure your CUDA Toolkit and cuBLAS are up-to-date.

Check Bazel Configuration: Double-check your Bazel configuration files (/.bazelrc and /.tf_configure.bazelrc) to ensure that they are correctly specifying the CUDA-related options and paths. Make sure they match your system's configuration.

Check Environment Variables: Verify that environment variables related to CUDA and TensorFlow are correctly set. For example, ensure that LD_LIBRARY_PATH includes the paths to CUDA libraries and TensorFlow.
Please Correct me If I am wrong @arjun988 Factually I have included the option -fPIC when compiling. It is very strange that it successfully compiled the non-debug version. As you can see that my CUDA version is 12.2, which version of TensorFlow do you think is compatible with this CUDA version? @terU3760 Actually TensorFlow and CUDA 12.2 are not officially compatible. TensorFlow 2.10 is compatible with CUDA 11.2 and cuDNN 8.1 for GPU support on Windows 7 (64 bit) or later Starting with TensorFlow 2.11, you need to install TensorFlow in WSL2 for GPU setup which requires Windows 10 19044 or higher However, some users have reported that they were able to use TensorFlow with CUDA 12.2 by using the latest NGC TF containers . You can check the compatibility over here [https://www.tensorflow.org/install/source#gpu](url) . Hi @terU3760 ,

Could you please confirm the tensorflow version you want to build. I can see it mentioned as 2.0 which is quiet old and we are not supporting it actively. Also you are using quiet older version of Bazel with entirely new version of CUDA 12.2 version which is not yet tested on any stable release so far.

Please find the attached test [configurations](https://www.tensorflow.org/install/source#gpu) on Linux and request you to test for latest TF versions with the specified build configurations only and if still have problem with that please let us know.

Thanks! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62016">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62016">No</a>

### Issue type

Build/Install

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

tf 2.0

### Custom code

No

### OS platform and distribution

Linux Ubuntu 20.04

### Mobile device

_No response_

### Python version

3.8.10

### Bazel version

1.16.0

### GCC/compiler version

9.4.0

### CUDA/cuDNN version

12.2

### GPU model and memory

GTX 1080 & RTX 2080

### Current behavior?

Reported the following error:
```
WARNING: The following configs were expanded more than once: [cuda, using_cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=120
INFO: Reading rc options for 'build' from /root/tensorflow/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /root/tensorflow/.bazelrc:
  'build' options: --apple_platform_type=macos --define framework_shared_object=true --define open_source_build=true --java_toolchain=//third_party/toolchains/java:tf_java_toolchain --host_java_toolchain=//third_party/toolchains/java:tf_java_toolchain --define=tensorflow_enable_mlir_generated_gpu_kernels=0 --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --noincompatible_prohibit_aapt1 --enable_platform_specific_config --config=short_logs --config=v2
INFO: Reading rc options for 'build' from /root/tensorflow/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=/usr/bin/python3 --action_env PYTHON_LIB_PATH=/usr/local/lib/python3.8/dist-packages --python_path=/usr/bin/python3 --config=xla --action_env CUDA_TOOLKIT_PATH=/usr/local/cuda-11.7 --action_env TF_CUDA_COMPUTE_CAPABILITIES=6.0,6.1,6.2,7.0,7.2,7.5 --action_env LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 --action_env GCC_HOST_COMPILER_PATH=/usr/bin/x86_64-linux-gnu-gcc-9 --config=cuda --action_env TF_CONFIGURE_IOS=0
INFO: Found applicable config definition build:short_logs in file /root/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /root/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:xla in file /root/tensorflow/.bazelrc: --define=with_xla_support=true
INFO: Found applicable config definition build:cuda in file /root/tensorflow/.bazelrc: --config=using_cuda --define=using_cuda_nvcc=true
INFO: Found applicable config definition build:using_cuda in file /root/tensorflow/.bazelrc: --define=using_cuda=true --action_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --define=tensorflow_enable_mlir_generated_gpu_kernels=1
INFO: Found applicable config definition build:cuda in file /root/tensorflow/.bazelrc: --config=using_cuda --define=using_cuda_nvcc=true
INFO: Found applicable config definition build:using_cuda in file /root/tensorflow/.bazelrc: --define=using_cuda=true --action_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --define=tensorflow_enable_mlir_generated_gpu_kernels=1
INFO: Found applicable config definition build:dbg in file /root/tensorflow/.bazelrc: --config=opt -c dbg --cxxopt -DTF_LITE_DISABLE_X86_NEON --copt -DDEBUG_BUILD
INFO: Found applicable config definition build:opt in file /root/tensorflow/.tf_configure.bazelrc: --copt=-Wno-sign-compare --host_copt=-Wno-sign-compare --define with_default_optimizations=true
INFO: Found applicable config definition build:linux in file /root/tensorflow/.bazelrc: --copt=-w --host_copt=-w --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++14 --host_cxxopt=-std=c++14 --config=dynamic_kernels
INFO: Found applicable config definition build:dynamic_kernels in file /root/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
INFO: Build options --copt and --cxxopt have changed, discarding analysis cache.
INFO: Analyzed target //tensorflow/tools/pip_package:build_pip_package (0 packages loaded, 32967 targets configured).
INFO: Found 1 target...
ERROR: /root/tensorflow/tensorflow/BUILD:724:1: Linking of rule '//tensorflow:libtensorflow_framework.so.2.4.4' failed (Exit 1)
/usr/bin/ld: bazel-out/k8-dbg/bin/tensorflow/stream_executor/cuda/libcublas_plugin.lo(cuda_blas.pic.o): relocation R_X86_64_PC32 against undefined symbol `_ZN15stream_executor3gpu12_GLOBAL__N_120CUDABlasLtMatmulPlan14kMaxBatchCountE' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 7241.142s, Critical Path: 2298.04s
INFO: 6729 processes: 6729 local.
FAILED: Build did NOT complete successfully
```

### Standalone code to reproduce the issue

```shell
git clone -b r2.0 https://github.com/tensorflow/tensorflow.git
bazel build --config=cuda --config=dbg --copt=-fPIC --cxxopt=-fPIC //tensorflow/tools/pip_package:build_pip_package
```


### Relevant log output

_No response_