undefined reference to `cudaGraphDebugDotPrint' when compiling TensorFlow 2.12
Hi @ujjwalnur ,

Request you to confirm the below things.

1. Command used for the build
2. Can you confirm whether you have followed the instructions [here](https://www.tensorflow.org/install/pip#step-by-step_instructions) mentioned in documentation for enabling GPU setup.
3. Also for Tf 2.12 version recommended CUDA is 11.8, cuDNN is 8.6. You can find the recommended and tested configurations [here](https://www.tensorflow.org/install/source#gpu).

Also please test by adding the option `--per_file_copt=//tensorflow/.*\.cc@-g,-O0` and let us know if it works.

Providing exact build command and steps followed may help us if above also won't work.

Thanks! > Hi @ujjwalnur ,
> 
> Request you to confirm the below things.
> 
>     1. Command used for the build
> 
>     2. Can you confirm whether you have followed the instructions [here](https://www.tensorflow.org/install/pip#step-by-step_instructions) mentioned in documentation for enabling GPU setup.
> 
>     3. Also for Tf 2.12 version recommended CUDA is 11.8, cuDNN is 8.6. You can find the recommended and tested configurations [here](https://www.tensorflow.org/install/source#gpu).
> 
> 
> Also please test by adding the option `--per_file_copt=//tensorflow/.*\.cc@-g,-O0` and let us know if it works.
> 
> Providing exact build command and steps followed may help us if above also won't work.
> 
> Thanks!

Command used for build : `bazel build --config=opt --config=cuda -k //tensorflow/tools/pip_package:build_pip_package`

I confirm that all the instructions were followed for enabling GPU setup.

After enabling the option` --per_file_copt=//tensorflow/.*\.cc@-g,-O0`  the code still does not link and gives the following error  -
```
ERROR: /tmp/tensorflow/tensorflow/BUILD:1219:21: Linking tensorflow/libtensorflow_cc.so.2.12.0 failed: (E
xit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/cr
osstool/clang/bin/crosstool_wrapper_driver_is_not_gcc @bazel-out/k8-opt/bin/tensorflow/libtensorflow_cc.s
o.2.12.0-2.params
/usr/bin/ld: bazel-out/k8-opt/bin/tensorflow/compiler/xla/stream_executor/cuda/libcuda_graph.pic.a(cuda_g
raph.pic.o): in function `stream_executor::gpu::CaptureCudaGraph(stream_executor::Stream*, absl::lts_2022
0623::AnyInvocable<tsl::Status ()>, cudaStreamCaptureMode)':
/proc/self/cwd/tensorflow/compiler/xla/stream_executor/cuda/cuda_graph.cc:136: undefined reference to `cu
daGraphDebugDotPrint'
bazel-out/k8-opt/bin/tensorflow/dtensor/mlir/libdtensor_mlir_passes.pic.lo(propagate_device_id_to_function_args.pic.o):(.debug_aranges+0x6): relocation truncated to fit: R_X86_64_32 against `.debug_info'
bazel-out/k8-opt/bin/tensorflow/dtensor/mlir/libdtensor_mlir_passes.pic.lo(restore_shape_inference.pic.o):(.debug_aranges+0x6): relocation truncated to fit: R_X86_64_32 against `.debug_info'
bazel-out/k8-opt/bin/tensorflow/dtensor/mlir/libdtensor_mlir_passes.pic.lo(set_default_sharding.pic.o):(.debug_aranges+0x6): relocation truncated to fit: R_X86_64_32 against `.debug_info'
bazel-out/k8-opt/bin/tensorflow/dtensor/mlir/libdtensor_mlir_passes.pic.lo(sparse_expansion.pic.o):(.debug_aranges+0x6): relocation truncated to fit: R_X86_64_32 against `.debug_info'
bazel-out/k8-opt/bin/tensorflow/dtensor/mlir/libdtensor_mlir_passes.pic.lo(spmd_expansion.pic.o):(.debug_aranges+0x6): relocation truncated to fit: R_X86_64_32 against `.debug_info'
bazel-out/k8-opt/bin/tensorflow/dtensor/mlir/libdtensor_mlir_passes.pic.lo(tpu_add_resource_device_attribute.pic.o):(.debug_aranges+0x6): relocation truncated to fit: R_X86_64_32 against `.debug_info'
bazel-out/k8-opt/bin/tensorflow/dtensor/mlir/libdtensor_mlir_passes.pic.lo(tpu_integration.pic.o):(.debug_aranges+0x6): relocation truncated to fit: R_X86_64_32 against `.debug_info'
bazel-out/k8-opt/bin/tensorflow/dtensor/mlir/libdtensor_mlir_passes.pic.lo(undo_merge_const_across_mesh.pic.o):(.debug_aranges+0x6): relocation truncated to fit: R_X86_64_32 against `.debug_info'
bazel-out/k8-opt/bin/tensorflow/dtensor/cc/liblayout_to_xla_sharding.pic.a(layout_to_xla_sharding.pic.o):(.debug_aranges+0x6): relocation truncated to fit: R_X86_64_32 against `.debug_info'
bazel-out/k8-opt/bin/tensorflow/dtensor/mlir/utils/libdtensor_mlir_passes_internal.pic.a(collective_lowering.pic.o):(.debug_aranges+0x6): relocation truncated to fit: R_X86_64_32 against `.debug_info'
bazel-out/k8-opt/bin/tensorflow/dtensor/mlir/utils/libdtensor_mlir_passes_internal.pic.a(dtensor_mlir_passes_internal.pic.o):(.debug_aranges+0x6): additional relocation overflows omitted from the output
collect2: error: ld returned 1 exit status
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1338.070s, Critical Path: 819.66s
INFO: 15964 processes: 247 internal, 15717 local.
FAILED: Build did NOT complete successfully
```


 I think that the following commit is responsible for the failed build : 

[de4440c2f8d0a84774afdef180d3ae4acab5398a](https://github.com/tensorflow/tensorflow/commit/de4440c2f8d0a84774afdef180d3ae4acab5398a) (For v2.12 branch)

The later commit [aae3467d8c5c8ae0a18dbfedecce7b50f2ac5ce7](https://github.com/tensorflow/tensorflow/commit/aae3467d8c5c8ae0a18dbfedecce7b50f2ac5ce7) adds a check for `CUDA version>=12000` but has not been put as part of v2.12 branch

Can you confirm that these changes need to be merged into v2.12 branch for a proper build ? 

I can confirm that I was able to build v2.11 successfully on CUDA 11.5 @ujjwalnur ,

Thanks for the information.We have observed build failures with `--config=cuda` and team already working on it.

I request you to provide the steps on how you have configured the CUDA and cuDNN paths.
 @SuryanarayanaY 
CUDA and cudnn paths were given as base paths to the custom installed paths.

These paths were setup using `configure.py` 

These paths are properly setup. As I have already said, for v2.11 I used the same paths without error as inall the versions since v2. 3 @SuryanarayanaY Any updates yet on this ? Even some technical explanations would be helpful because then I can take a look into this problem. Since I am a newbie to TF source code, a little help could go a long way. @ujjwalnur @SuryanarayanaY 
I have the same problem.
tensorflow_version:2.12.0
cuda_version:11.6
[tensorflow-2.12.0/tensorflow/compiler/xla/stream_executor/cuda/cuda_graph.cc](https://github.com/tensorflow/tensorflow/blob/v2.12.0/tensorflow/compiler/xla/stream_executor/cuda/cuda_graph.cc)
 Hi @ujjwalnur ,

Apologies for the delayed response. Could you please confirm the path setting instructions followed by you to enable CUDA and cuDNN paths. Could you please check the Nvidia instructions [here](https://docs.nvidia.com/cuda/cuda-quick-start-guide/index.html#ubuntu) for same and try the build and let us know the outcome. 

Thanks! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60689">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60689">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Build/Install

### Have you reproduced the bug with TF nightly?

No

### Source

source

### Tensorflow Version

2.12

### Custom Code

No

### OS Platform and Distribution

Ubuntu 20.04

### Mobile device

_No response_

### Python version

3.9

### Bazel version

5.3.0

### GCC/Compiler version

9.4.0

### CUDA/cuDNN version

11.5/8.3

### GPU model and memory

NVidia RTX A5000

### Current Behaviour?

The error comes during linking. Please see the log output.


### Standalone code to reproduce the issue

```shell
I just use the standard bazel build command on TensorFlow website to compile.
```


### Relevant log output

```shell
ERROR: /tmp/tensorflow/tensorflow/BUILD:1219:21: Linking tensorflow/libtensorflow_cc.so.2.12.0 failed: (E
xit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command
  (cd /home/ujjwal/.cache/bazel/_bazel_ujjwal/e5cce820cc082410b4fcc604db349066/execroot/org_tensorflow &&
 \
  exec env - \
    CUDA_TOOLKIT_PATH=/home/ujjwal/softwares/cuda-11.5 \
    GCC_HOST_COMPILER_PATH=/usr/bin/x86_64-linux-gnu-gcc-9 \
    LD_LIBRARY_PATH=:/home/ujjwal/softwares/cuda-11.5/lib64:/home/ujjwal/softwares/nccl_2.11.4-1+cuda11.5
_x86_64/lib:/home/ujjwal/softwares/TensorRT-8.2.1.8/lib:/home/ujjwal/softwares/zlib/lib:/home/ujjwal/soft
wares/gettext/lib:/home/ujjwal/softwares/curl/lib:/home/ujjwal/softwares/openssl/lib64:/home/ujjwal/softwares/cudnn-8.3.1.22/lib:/home/ujjwal/softwares/cuda-11.5/extras/CUPTI/lib64 \
    PATH=/home/ujjwal/.cargo/bin:/home/ujjwal/anaconda3/envs/tf2.12/bin:/home/ujjwal/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/ujjwal/softwares/cuda-11.5/bin:/home/ujjwal/softwares/bazel-5.3.0/bin:/home/ujjwal/softwares/TensorRT-8.2.1.8/bin:/home/ujjwal/softwares/gettext/bin:/home/ujjwal/softwares/git/bin:/home/ujjwal/softwares/curl/bin:/home/ujjwal/softwares/openssl/bin:/home/ujjwal/softwares/neovim \
    PWD=/proc/self/cwd \
    PYTHON_BIN_PATH=/home/ujjwal/anaconda3/envs/tf2.12/bin/python \
    PYTHON_LIB_PATH=/home/ujjwal/anaconda3/envs/tf2.12/lib/python3.9/site-packages \
    TF2_BEHAVIOR=1 \
    TF_CUDA_COMPUTE_CAPABILITIES=8.6,8.6,8.6,8.6,8.6,8.6,8.6,8.6 \
    TF_CUDA_PATHS=/home/ujjwal/softwares/nccl_2.11.4-1+cuda11.5_x86_64,/home/ujjwal/softwares/cudnn-8.3.1.22,/home/ujjwal/softwares/cuda-11.5,/home/ujjwal/softwares/TensorRT-8.2.1.8 \
    TF_CUDA_VERSION=11.5 \
    TF_CUDNN_VERSION=8.3 \
    TF_NCCL_VERSION=2.11.4 \
    TF_TENSORRT_VERSION=8.2 \
  external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc @bazel-out/k8-opt/bin/tensorflow/libtensorflow_cc.so.2.12.0-2.params)
# Configuration: c6f4dad0752984ef77f185453bec6416ee82f657120171617f8b93e228363e95
# Execution platform: @local_execution_config_platform//:platform
/usr/bin/ld: bazel-out/k8-opt/bin/tensorflow/compiler/xla/stream_executor/cuda/libcuda_graph.pic.a(cuda_graph.pic.o): in function `stream_executor::gpu::CaptureCudaGraph(stream_executor::Stream*, absl::lts_20220623::AnyInvocable<tsl::Status ()>, cudaStreamCaptureMode)':
cuda_graph.cc:(.text._ZN15stream_executor3gpu16CaptureCudaGraphEPNS_6StreamEN4absl12lts_2022062312AnyInvocableIFN3tsl6StatusEvEEE21cudaStreamCaptureMode+0x520): undefined reference to `cudaGraphDebugDotPrint'
collect2: error: ld returned 1 exit status
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 1323.055s, Critical Path: 476.12s
INFO: 24057 processes: 158 internal, 23899 local.
FAILED: Build did NOT complete successfully
```
</details>