Tensorflow 2.12 bazel failed!Cuda11.2+cuDNN8.1+Tensorrt7.2 under Ubuntu2004
@mihaimaruseac, same here. I am trying to build TensorFlow 2.12 in docker:

```
sudo docker run --gpus all -it -w /tensorflow -v /home/vishnya/tensorflow:/tensorflow -v $PWD:/mnt -e HOST_PERMS="\\((id -u):\\)(id -g)" tensorflow/tensorflow:devel-gpu bash
bazel build --config=cuda //tensorflow/tools/pip_package:build_pip_package
``` When building from master, please always give the commit, given TF moves fast forward.

Does it reproduce if you switch to a newer commit? Sorry, I misread the link to the `.bazelrc` in master branch.

Can you try a `bazel clean --expunge` and then build again?

Adding @angerson and @MichaelHudgins for the Docker stuff too. > When building from master, please always give the commit, given TF moves fast forward.
> 
> Does it reproduce if you switch to a newer commit?

@mihaimaruseac, TensorFlow fails to build from `r2.12` branch. I haven't tried master, because I try to locally rebuild 2.12 release.

> Can you try a `bazel clean --expunge` and then build again?

I'll try it. > Can you try a `bazel clean --expunge` and then build again?

@mihaimaruseac, it didn't help. The same error:

```
ERROR: /tensorflow/tensorflow/compiler/xla/stream_executor/cuda/BUILD:519:11: Compiling tensorflow/compiler/xla/stream_executor/cuda/cuda_graph.cc failed: (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF bazel-out/k8-opt/bin/tensorflow/compiler/xla/stream_executor/cuda/_objs/cuda_graph/cuda_graph.pic.d ... (remaining 108 arguments skipped)
tensorflow/compiler/xla/stream_executor/cuda/cuda_graph.cc: In function ‘tsl::StatusOr<stream_executor::gpu::OwnedCudaGraph> stream_executor::gpu::CaptureCudaGraph(stream_executor::Stream*, absl::lts_20220623::AnyInvocable<tsl::Status()>, cudaStreamCaptureMode)’:
tensorflow/compiler/xla/stream_executor/cuda/cuda_graph.cc:135:21: error: ‘cudaGraphDebugDotFlagsVerbose’ was not declared in this scope
  135 |         int flags = cudaGraphDebugDotFlagsVerbose;
      |                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tensorflow/compiler/xla/stream_executor/cuda/cuda_graph.cc:136:24: error: ‘cudaGraphDebugDotPrint’ was not declared in this scope
  136 |         if (auto err = cudaGraphDebugDotPrint(graph, file.c_str(), flags);
      |                        ^~~~~~~~~~~~~~~~~~~~~~
Target //tensorflow/tools/pip_package:build_pip_package failed to build
``` Could it be because 2.12 uses cuDNN 8.6, Cuda 11.8? > Could it be because 2.12 uses cuDNN 8.6, Cuda 11.8?

```
nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Feb_14_21:12:58_PST_2021
Cuda compilation tools, release 11.2, V11.2.152
Build cuda_11.2.r11.2/compiler.29618528_0
```

Hmm, maybe I am using the wrong docker container. What's the right docker container with fresh cuda to build 2.12? Tagging @angerson  Now I am trying the following docker (at least it has correct CUDA version):

```
sudo docker run --rm --gpus all -it -w /tensorflow -v /home/vishnya/tensorflow:/tensorflow -v $PWD:/mnt -e HOST_PERMS="\\((id -u):\\)(id -g)" tensorflow/build:2.12-python3.8 bash
bazel build --config=cuda //tensorflow/tools/pip_package:build_pip_package
``` As for docker containers using the tensorflow/build ones would be the correct ones.  See: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/tools/tf_sig_build_dockerfiles.

I agree that it likely is the mismatch on the original issue as well.  The root level .bazelrc should get an update in the next few days as clang is migrated.   Until then https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/tf_sig_build_dockerfiles/devel.usertools/gpu_gcc.bazelrc has the bazelrc values being used in TF's CI. 

 Thanks Michael & Mihai. Those instructions Michael linked should work.

I'll have to look into updating the https://www.tensorflow.org/install/source#docker_linux_builds docs. I guess they never got changed, as the SIG Build containers were not initially planned to be official. I succeeded building TensorFlow 2.12 in `tensorflow/build:2.12-python3.8` docker. Thank you all.  @SweetVishnya @angerson @MichaelHudgins @mihaimaruseac 
The relevant documents [build from source](https://www.tensorflow.org/install/source) have been updated to the latest version.well done!
![image](https://user-images.githubusercontent.com/27951761/233656393-d56e7f19-af5f-4c00-94ba-597f4228a417.png)

BTW:
bazel.rc [TensorRT 7 for CUDA 11.1 is compatible with CUDA 11.2](https://github.com/tensorflow/tensorflow/blob/master/.bazelrc#L492)
CUDA 11.2 must be [11.2 update 1](https://developer.nvidia.com/cuda-toolkit-archive),not 11.2 update 2, refer to here [tensorrt-723/support-matrix](https://docs.nvidia.com/deeplearning/tensorrt/archives/tensorrt-723/support-matrix/index.html)

nvrtc is deliver inside the CUDA toolkit, you can also install it separate, I find the links in:

ubuntu1804
https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-nvrtc-11-1_11.1.105-1_amd64.deb
https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-nvrtc-dev-11-1_11.1.105-1_amd64.deb

ubuntu2004
https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu2004/x86_64/cuda-nvrtc-11-1_11.1.105-1_amd64.deb
https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu2004/x86_64/cuda-nvrtc-dev-11-1_11.1.105-1_amd64.deb

ubuntu2204
https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu2204/x86_64/cuda-nvrtc-11-1_11.1.105-1_amd64.deb https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu2204/x86_64/cuda-nvrtc-dev-11-1_11.1.105-1_amd64.deb In this case, we can close this? Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60362">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60362">No</a>
 And，I sucessed  to build the latest 2.12 version with Cuda11.8+cuDNN8.6+Tensorrt**8.5.2.2**+JAX+Flax under Ubuntu2004 at last.

you will continue to meet new "DataType kFP8" error as shown in the following information  if you use the tensorrt version between [8.5.3](https://docs.nvidia.com/deeplearning/tensorrt/archives/tensorrt-853/api/c_api/namespacenvinfer1.html#a83aed11a1c160f30dcd13809678bdd29) and [8.6.0](https://docs.nvidia.com/deeplearning/tensorrt/archives/tensorrt-860-ea/api/c_api/namespacenvinfer1.html#a83aed11a1c160f30dcd13809678bdd29) of TensorRT. (API Reference: [8.5.3](https://docs.nvidia.com/deeplearning/tensorrt/archives/tensorrt-853/api/c_api/namespacenvinfer1.html#a83aed11a1c160f30dcd13809678bdd29), [8.6.0](https://docs.nvidia.com/deeplearning/tensorrt/archives/tensorrt-860-ea/api/c_api/namespacenvinfer1.html#a83aed11a1c160f30dcd13809678bdd29))

Execution platform: @local_execution_config_platform//:platform
tensorflow/compiler/tf2tensorrt/convert/weights.cc: In member function ‘size_t tensorflow::tensorrt::convert::TRT_ShapedWeights::size_bytes() const’:
tensorflow/compiler/tf2tensorrt/convert/weights.cc:61:10: error: enumeration value ‘kFP8’ not handled in switch [-Werror=switch]
   61 |   switch (type_) {
      |          ^
cc1plus: some warnings being treated as errors
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 1276.053s, Critical Path: 83.16s
INFO: 8084 processes: 3883 internal, 4201 local.
FAILED: Build did NOT complete successfully
 Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60362">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60362">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Build/Install

### Have you reproduced the bug with TF nightly?

No

### Source

source

### Tensorflow Version

tf2.12

### Custom Code

Yes

### OS Platform and Distribution

Ubuntu2004

### Mobile device

_No response_

### Python version

3.8

### Bazel version

5.3

### GCC/Compiler version

9.4

### CUDA/cuDNN version

CUDA11.2/cuDNN8.1

### GPU model and memory

_No response_

##Current Behaviour?

TensorRT Version: 7.2.3.4 CUDA Version: 11.2 CUDNN Version: 8.1.1.33-1 Operating System: Ubuntu-20.04

https://github.com/tensorflow/tensorflow/blob/master/.bazelrc

Bazel build latest 2.12 source code failed:


##Standalone code to reproduce the issue

```shell
# Configuration: 605fee2c8aa1b68d8dcb7abeac1c0e77048ffd56710c9ebb1fee2729853263d3
# Execution platform: @local_execution_config_platform//:platform
tensorflow/compiler/xla/stream_executor/cuda/cuda_graph.cc: In function ‘tsl::StatusOr<stream_executor::gpu::OwnedCudaGraph> stream_executor::gpu::CaptureCudaGraph(stream_executor::Stream*, absl::lts_20220623::AnyInvocable<tsl::Status()>, cudaStreamCaptureMode)’:
tensorflow/compiler/xla/stream_executor/cuda/cuda_graph.cc:135:21: **error**: ‘cudaGraphDebugDotFlagsVerbose’ was not declared in this scope
  135 |         int flags = cudaGraphDebugDotFlagsVerbose;
      |                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tensorflow/compiler/xla/stream_executor/cuda/cuda_graph.cc:136:24: **error**: ‘cudaGraphDebugDotPrint’ was not declared in this scope
  136 |         if (auto err = cudaGraphDebugDotPrint(graph, file.c_str(), flags);
      |                        ^~~~~~~~~~~~~~~~~~~~~~
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 11358.490s, Critical Path: 270.65s
INFO: 26270 processes: 9762 internal, 16508 local.
FAILED: Build did NOT complete successfully
```


### Relevant log output

_No response_</details>