Failure in building with GPU support from source specifically in 2.7.0
after some debugging, i found out that it might have something to do with find_cuda_config.py not able to locate `/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcusolver.so.11` so i manually specified `cusolver_version = "11"`. Now my cuda config looks like this `struct(compute_capabilities = ["compute_86"], config = {"cublas_include_dir": "/usr/local/cuda-11.8/targets/x86_64-linux/include", "cublas_library_dir": "/usr/local/cuda-11.8/targets/x86_64-linux/lib", "cublas_version": "11.11.3", "cuda_binary_dir": "/usr/local/cuda-11.8/bin", "cuda_include_dir": "/usr/local/cuda-11.8/targets/x86_64-linux/include", "cuda_library_dir": "/usr/local/cuda-11.8/targets/x86_64-linux/lib", "cudnn_include_dir": "/usr/include", "cudnn_library_dir": "/usr/lib/x86_64-linux-gnu", "cufft_include_dir": "/usr/local/cuda-11.8/targets/x86_64-linux/include", "cufft_library_dir": "/usr/local/cuda-11.8/targets/x86_64-linux/lib", "cufft_version": "10.9.0", "cupti_include_dir": "/usr/local/cuda-11.8/targets/x86_64-linux/include", "cupti_library_dir": "/usr/local/cuda-11.8/targets/x86_64-linux/lib", "curand_include_dir": "/usr/local/cuda-11.8/targets/x86_64-linux/include", "curand_library_dir": "/usr/local/cuda-11.8/targets/x86_64-linux/lib", "curand_version": "10.3.0", "cusolver_include_dir": "/usr/local/cuda-11.8/targets/x86_64-linux/include", "cusolver_library_dir": "/usr/local/cuda-11.8/targets/x86_64-linux/lib", "cusolver_version": "11", "cusparse_include_dir": "/usr/local/cuda-11.8/targets/x86_64-linux/include", "cusparse_library_dir": "/usr/local/cuda-11.8/targets/x86_64-linux/lib", "cusparse_version": "11.7.5", "nvvm_library_dir": "/usr/local/cuda-11.8/nvvm/libdevice"}, cpu_value = "Linux", cublas_version = "11", cuda_toolkit_path = "/usr/local/cuda-11.8", cuda_version = "11.8", cuda_version_major = "11", cudart_version = "11.0", cudnn_version = "8", cufft_version = "10", curand_version = "10", cusolver_version = "11", cusparse_version = "11")`

And it managed to start compiling but in the end when `executing genrule //tensorflow/python/keras/api:keras_python_api_gen_compat_v1` it gives error as well 
```ERROR: /home/haolan/tf-gpu3/tensorflow/python/keras/api/BUILD:133:19: Executing genrule //tensorflow/python/keras/api:keras_python_api_gen_compat_v1 failed (Exit 1): bash failed: error executing command
  (cd /home/haolan/.cache/bazel/_bazel_haolan/4841e33fe850472b4b360662c0c7abcc/execroot/org_tensorflow && \
  exec env - \
    CUDA_TOOLKIT_PATH=/usr/local/cuda-11.8 \
    GCC_HOST_COMPILER_PATH=/usr/bin/x86_64-linux-gnu-gcc-11 \
    LD_LIBRARY_PATH=/usr/local/cuda-11.8/lib64: \
    PATH=/home/haolan/test/bin:/usr/local/cuda-11.8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
    PYTHON_BIN_PATH=/home/haolan/test/bin/python \
    PYTHON_LIB_PATH=/home/haolan/test/lib/python3.8/site-packages \
    TF2_BEHAVIOR=1 \
    TF_CUDA_COMPUTE_CAPABILITIES=8.6 \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel-out/k8-dbg/bin/tensorflow/python/keras/api/create_tensorflow.python_api_keras_python_api_gen_compat_v1  --apidir=bazel-out/k8-dbg/bin/tensorflow/python/keras/api_v1/ --apiname=keras --apiversion=1  --loading=default --packages=tensorflow.python,tensorflow.python.keras,tensorflow.python.keras.activations,tensorflow.python.keras.applications.densenet,tensorflow.python.keras.applications.efficientnet,tensorflow.python.keras.applications.imagenet_utils,tensorflow.python.keras.applications.inception_resnet_v2,tensorflow.python.keras.applications.inception_v3,tensorflow.python.keras.applications.mobilenet,tensorflow.python.keras.applications.mobilenet_v2,tensorflow.python.keras.applications.mobilenet_v3,tensorflow.python.keras.applications.nasnet,tensorflow.python.keras.applications.resnet,tensorflow.python.keras.applications.resnet_v2,tensorflow.python.keras.applications.vgg16,tensorflow.python.keras.applications.vgg19,tensorflow.python.keras.applications.xception,tensorflow.python.keras.backend,tensorflow.python.keras.backend_config,tensorflow.python.keras.callbacks,tensorflow.python.keras.callbacks_v1,tensorflow.python.keras.constraints,tensorflow.python.keras.datasets.boston_housing,tensorflow.python.keras.datasets.cifar10,tensorflow.python.keras.datasets.cifar100,tensorflow.python.keras.datasets.fashion_mnist,tensorflow.python.keras.datasets.imdb,tensorflow.python.keras.datasets.mnist,tensorflow.python.keras.datasets.reuters,tensorflow.python.keras.engine.base_layer,tensorflow.python.keras.engine.data_adapter,tensorflow.python.keras.engine.input_layer,tensorflow.python.keras.engine.input_spec,tensorflow.python.keras.engine.sequential,tensorflow.python.keras.engine.training,tensorflow.python.keras.estimator,tensorflow.python.keras.feature_column.sequence_feature_column,tensorflow.python.keras.initializers,tensorflow.python.keras.initializers.initializers_v1,tensorflow.python.keras.initializers.initializers_v2,tensorflow.python.keras.layers.advanced_activations,tensorflow.python.keras.layers.convolutional,tensorflow.python.keras.layers.convolutional_recurrent,tensorflow.python.keras.layers.core,tensorflow.python.keras.layers.cudnn_recurrent,tensorflow.python.keras.layers.dense_attention,tensorflow.python.keras.layers.embeddings,tensorflow.python.keras.layers.legacy_rnn.rnn_cell_impl,tensorflow.python.keras.layers.local,tensorflow.python.keras.layers.merge,tensorflow.python.keras.layers.noise,tensorflow.python.keras.layers.normalization.batch_normalization,tensorflow.python.keras.layers.normalization.batch_normalization_v1,tensorflow.python.keras.layers.normalization.layer_normalization,tensorflow.python.keras.layers.preprocessing,tensorflow.python.keras.layers.pooling,tensorflow.python.keras.layers.recurrent,tensorflow.python.keras.layers.recurrent_v2,tensorflow.python.keras.layers.serialization,tensorflow.python.keras.layers.wrappers,tensorflow.python.keras.legacy_tf_layers.base,tensorflow.python.keras.legacy_tf_layers.convolutional,tensorflow.python.keras.legacy_tf_layers.core,tensorflow.python.keras.legacy_tf_layers.normalization,tensorflow.python.keras.legacy_tf_layers.pooling,tensorflow.python.keras.losses,tensorflow.python.keras.metrics,tensorflow.python.keras.mixed_precision.get_layer_policy,tensorflow.python.keras.mixed_precision.loss_scale_optimizer,tensorflow.python.keras.mixed_precision.policy,tensorflow.python.keras.models,tensorflow.python.keras.optimizer_v2.adadelta,tensorflow.python.keras.optimizer_v2.adagrad,tensorflow.python.keras.optimizer_v2.adam,tensorflow.python.keras.optimizer_v2.adamax,tensorflow.python.keras.optimizer_v2.ftrl,tensorflow.python.keras.optimizer_v2.gradient_descent,tensorflow.python.keras.optimizer_v2.learning_rate_schedule,tensorflow.python.keras.optimizer_v2.nadam,tensorflow.python.keras.optimizer_v2.optimizer_v2,tensorflow.python.keras.optimizer_v2.rmsprop,tensorflow.python.keras.optimizers,tensorflow.python.keras.premade.linear,tensorflow.python.keras.premade.wide_deep,tensorflow.python.keras.preprocessing.image,tensorflow.python.keras.preprocessing.sequence,tensorflow.python.keras.preprocessing.text,tensorflow.python.keras.regularizers,tensorflow.python.keras.saving.model_config,tensorflow.python.keras.saving.save,tensorflow.python.keras.saving.saved_model_experimental,tensorflow.python.keras.utils.data_utils,tensorflow.python.keras.utils.generic_utils,tensorflow.python.keras.utils.io_utils,tensorflow.python.keras.utils.layer_utils,tensorflow.python.keras.utils.losses_utils,tensorflow.python.keras.utils.multi_gpu_utils,tensorflow.python.keras.utils.np_utils,tensorflow.python.keras.utils.tf_utils,tensorflow.python.keras.utils.vis_utils,tensorflow.python.keras.wrappers.scikit_learn --output_package=tensorflow.python.keras.api._v1 --use_relative_imports=True bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/__internal__/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/__internal__/legacy/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/__internal__/legacy/layers/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/__internal__/legacy/layers/experimental/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/__internal__/legacy/rnn_cell/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/activations/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/densenet/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/efficientnet/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/imagenet_utils/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/inception_resnet_v2/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/inception_v3/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/mobilenet/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/mobilenet_v2/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/mobilenet_v3/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/nasnet/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/resnet/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/resnet_v2/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/resnet50/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/vgg16/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/vgg19/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/xception/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/backend/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/callbacks/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/callbacks/experimental/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/constraints/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/datasets/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/datasets/boston_housing/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/datasets/cifar10/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/datasets/cifar100/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/datasets/fashion_mnist/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/datasets/imdb/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/datasets/mnist/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/datasets/reuters/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/estimator/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/experimental/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/initializers/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/layers/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/layers/experimental/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/layers/experimental/preprocessing/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/losses/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/metrics/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/mixed_precision/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/mixed_precision/experimental/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/models/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/optimizers/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/optimizers/schedules/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/premade/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/preprocessing/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/preprocessing/image/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/preprocessing/sequence/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/preprocessing/text/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/regularizers/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/utils/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/wrappers/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/wrappers/scikit_learn/__init__.py')
Execution platform: @local_execution_config_platform//:platform
Traceback (most recent call last):
  File "/home/haolan/.cache/bazel/_bazel_haolan/4841e33fe850472b4b360662c0c7abcc/execroot/org_tensorflow/bazel-out/k8-dbg/bin/tensorflow/python/keras/api/create_tensorflow.python_api_keras_python_api_gen_compat_v1.runfiles/org_tensorflow/tensorflow/python/pywrap_tensorflow.py", line 64, in <module>
    from tensorflow.python._pywrap_tensorflow_internal import *
ImportError: /home/haolan/.cache/bazel/_bazel_haolan/4841e33fe850472b4b360662c0c7abcc/execroot/org_tensorflow/bazel-out/k8-dbg/bin/tensorflow/python/keras/api/create_tensorflow.python_api_keras_python_api_gen_compat_v1.runfiles/org_tensorflow/tensorflow/python/_pywrap_tensorflow_internal.so: undefined symbol: _ZN4mlir2TF6detail25ResourceAliasAnalysisInfo18kMaxResourceTypeIdE

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/haolan/.cache/bazel/_bazel_haolan/4841e33fe850472b4b360662c0c7abcc/execroot/org_tensorflow/bazel-out/k8-dbg/bin/tensorflow/python/keras/api/create_tensorflow.python_api_keras_python_api_gen_compat_v1.runfiles/org_tensorflow/tensorflow/python/tools/api/generator/create_python_api.py", line 26, in <module>
    from tensorflow.python.tools.api.generator import doc_srcs
  File "/home/haolan/.cache/bazel/_bazel_haolan/4841e33fe850472b4b360662c0c7abcc/execroot/org_tensorflow/bazel-out/k8-dbg/bin/tensorflow/python/keras/api/create_tensorflow.python_api_keras_python_api_gen_compat_v1.runfiles/org_tensorflow/tensorflow/python/__init__.py", line 40, in <module>
    from tensorflow.python import pywrap_tensorflow as _pywrap_tensorflow
  File "/home/haolan/.cache/bazel/_bazel_haolan/4841e33fe850472b4b360662c0c7abcc/execroot/org_tensorflow/bazel-out/k8-dbg/bin/tensorflow/python/keras/api/create_tensorflow.python_api_keras_python_api_gen_compat_v1.runfiles/org_tensorflow/tensorflow/python/pywrap_tensorflow.py", line 79, in <module>
    raise ImportError(
ImportError: Traceback (most recent call last):
  File "/home/haolan/.cache/bazel/_bazel_haolan/4841e33fe850472b4b360662c0c7abcc/execroot/org_tensorflow/bazel-out/k8-dbg/bin/tensorflow/python/keras/api/create_tensorflow.python_api_keras_python_api_gen_compat_v1.runfiles/org_tensorflow/tensorflow/python/pywrap_tensorflow.py", line 64, in <module>
    from tensorflow.python._pywrap_tensorflow_internal import *
ImportError: /home/haolan/.cache/bazel/_bazel_haolan/4841e33fe850472b4b360662c0c7abcc/execroot/org_tensorflow/bazel-out/k8-dbg/bin/tensorflow/python/keras/api/create_tensorflow.python_api_keras_python_api_gen_compat_v1.runfiles/org_tensorflow/tensorflow/python/_pywrap_tensorflow_internal.so: undefined symbol: _ZN4mlir2TF6detail25ResourceAliasAnalysisInfo18kMaxResourceTypeIdE


Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/errors for some common causes and solutions.
If you need help, create an issue at https://github.com/tensorflow/tensorflow/issues and include the entire stack trace above this error message.
Target //tensorflow/tools/pip_package:build_pip_package failed to build
ERROR: /home/haolan/tf-gpu3/tensorflow/lite/python/BUILD:63:10 Executing genrule //tensorflow/python/keras/api:keras_python_api_gen_compat_v1 failed (Exit 1): bash failed: error executing command
  (cd /home/haolan/.cache/bazel/_bazel_haolan/4841e33fe850472b4b360662c0c7abcc/execroot/org_tensorflow && \
  exec env - \
    CUDA_TOOLKIT_PATH=/usr/local/cuda-11.8 \
    GCC_HOST_COMPILER_PATH=/usr/bin/x86_64-linux-gnu-gcc-11 \
    LD_LIBRARY_PATH=/usr/local/cuda-11.8/lib64: \
    PATH=/home/haolan/test/bin:/usr/local/cuda-11.8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
    PYTHON_BIN_PATH=/home/haolan/test/bin/python \
    PYTHON_LIB_PATH=/home/haolan/test/lib/python3.8/site-packages \
    TF2_BEHAVIOR=1 \
    TF_CUDA_COMPUTE_CAPABILITIES=8.6 \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel-out/k8-dbg/bin/tensorflow/python/keras/api/create_tensorflow.python_api_keras_python_api_gen_compat_v1  --apidir=bazel-out/k8-dbg/bin/tensorflow/python/keras/api_v1/ --apiname=keras --apiversion=1  --loading=default --packages=tensorflow.python,tensorflow.python.keras,tensorflow.python.keras.activations,tensorflow.python.keras.applications.densenet,tensorflow.python.keras.applications.efficientnet,tensorflow.python.keras.applications.imagenet_utils,tensorflow.python.keras.applications.inception_resnet_v2,tensorflow.python.keras.applications.inception_v3,tensorflow.python.keras.applications.mobilenet,tensorflow.python.keras.applications.mobilenet_v2,tensorflow.python.keras.applications.mobilenet_v3,tensorflow.python.keras.applications.nasnet,tensorflow.python.keras.applications.resnet,tensorflow.python.keras.applications.resnet_v2,tensorflow.python.keras.applications.vgg16,tensorflow.python.keras.applications.vgg19,tensorflow.python.keras.applications.xception,tensorflow.python.keras.backend,tensorflow.python.keras.backend_config,tensorflow.python.keras.callbacks,tensorflow.python.keras.callbacks_v1,tensorflow.python.keras.constraints,tensorflow.python.keras.datasets.boston_housing,tensorflow.python.keras.datasets.cifar10,tensorflow.python.keras.datasets.cifar100,tensorflow.python.keras.datasets.fashion_mnist,tensorflow.python.keras.datasets.imdb,tensorflow.python.keras.datasets.mnist,tensorflow.python.keras.datasets.reuters,tensorflow.python.keras.engine.base_layer,tensorflow.python.keras.engine.data_adapter,tensorflow.python.keras.engine.input_layer,tensorflow.python.keras.engine.input_spec,tensorflow.python.keras.engine.sequential,tensorflow.python.keras.engine.training,tensorflow.python.keras.estimator,tensorflow.python.keras.feature_column.sequence_feature_column,tensorflow.python.keras.initializers,tensorflow.python.keras.initializers.initializers_v1,tensorflow.python.keras.initializers.initializers_v2,tensorflow.python.keras.layers.advanced_activations,tensorflow.python.keras.layers.convolutional,tensorflow.python.keras.layers.convolutional_recurrent,tensorflow.python.keras.layers.core,tensorflow.python.keras.layers.cudnn_recurrent,tensorflow.python.keras.layers.dense_attention,tensorflow.python.keras.layers.embeddings,tensorflow.python.keras.layers.legacy_rnn.rnn_cell_impl,tensorflow.python.keras.layers.local,tensorflow.python.keras.layers.merge,tensorflow.python.keras.layers.noise,tensorflow.python.keras.layers.normalization.batch_normalization,tensorflow.python.keras.layers.normalization.batch_normalization_v1,tensorflow.python.keras.layers.normalization.layer_normalization,tensorflow.python.keras.layers.preprocessing,tensorflow.python.keras.layers.pooling,tensorflow.python.keras.layers.recurrent,tensorflow.python.keras.layers.recurrent_v2,tensorflow.python.keras.layers.serialization,tensorflow.python.keras.layers.wrappers,tensorflow.python.keras.legacy_tf_layers.base,tensorflow.python.keras.legacy_tf_layers.convolutional,tensorflow.python.keras.legacy_tf_layers.core,tensorflow.python.keras.legacy_tf_layers.normalization,tensorflow.python.keras.legacy_tf_layers.pooling,tensorflow.python.keras.losses,tensorflow.python.keras.metrics,tensorflow.python.keras.mixed_precision.get_layer_policy,tensorflow.python.keras.mixed_precision.loss_scale_optimizer,tensorflow.python.keras.mixed_precision.policy,tensorflow.python.keras.models,tensorflow.python.keras.optimizer_v2.adadelta,tensorflow.python.keras.optimizer_v2.adagrad,tensorflow.python.keras.optimizer_v2.adam,tensorflow.python.keras.optimizer_v2.adamax,tensorflow.python.keras.optimizer_v2.ftrl,tensorflow.python.keras.optimizer_v2.gradient_descent,tensorflow.python.keras.optimizer_v2.learning_rate_schedule,tensorflow.python.keras.optimizer_v2.nadam,tensorflow.python.keras.optimizer_v2.optimizer_v2,tensorflow.python.keras.optimizer_v2.rmsprop,tensorflow.python.keras.optimizers,tensorflow.python.keras.premade.linear,tensorflow.python.keras.premade.wide_deep,tensorflow.python.keras.preprocessing.image,tensorflow.python.keras.preprocessing.sequence,tensorflow.python.keras.preprocessing.text,tensorflow.python.keras.regularizers,tensorflow.python.keras.saving.model_config,tensorflow.python.keras.saving.save,tensorflow.python.keras.saving.saved_model_experimental,tensorflow.python.keras.utils.data_utils,tensorflow.python.keras.utils.generic_utils,tensorflow.python.keras.utils.io_utils,tensorflow.python.keras.utils.layer_utils,tensorflow.python.keras.utils.losses_utils,tensorflow.python.keras.utils.multi_gpu_utils,tensorflow.python.keras.utils.np_utils,tensorflow.python.keras.utils.tf_utils,tensorflow.python.keras.utils.vis_utils,tensorflow.python.keras.wrappers.scikit_learn --output_package=tensorflow.python.keras.api._v1 --use_relative_imports=True bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/__internal__/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/__internal__/legacy/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/__internal__/legacy/layers/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/__internal__/legacy/layers/experimental/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/__internal__/legacy/rnn_cell/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/activations/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/densenet/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/efficientnet/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/imagenet_utils/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/inception_resnet_v2/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/inception_v3/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/mobilenet/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/mobilenet_v2/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/mobilenet_v3/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/nasnet/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/resnet/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/resnet_v2/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/resnet50/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/vgg16/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/vgg19/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/applications/xception/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/backend/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/callbacks/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/callbacks/experimental/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/constraints/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/datasets/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/datasets/boston_housing/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/datasets/cifar10/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/datasets/cifar100/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/datasets/fashion_mnist/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/datasets/imdb/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/datasets/mnist/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/datasets/reuters/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/estimator/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/experimental/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/initializers/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/layers/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/layers/experimental/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/layers/experimental/preprocessing/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/losses/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/metrics/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/mixed_precision/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/mixed_precision/experimental/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/models/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/optimizers/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/optimizers/schedules/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/premade/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/preprocessing/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/preprocessing/image/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/preprocessing/sequence/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/preprocessing/text/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/regularizers/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/utils/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/wrappers/__init__.py bazel-out/k8-dbg/bin/tensorflow/python/keras/api/_v1/keras/wrappers/scikit_learn/__init__.py')
Execution platform: @local_execution_config_platform//:platform``` issue resolved by adding flag  `--cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0"` during build Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62663">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62663">No</a>

### Issue type

Build/Install

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

2.7.0

### Custom code

No

### OS platform and distribution

Ubuntu 22.04

### Mobile device

_No response_

### Python version

3.8

### Bazel version

3.7.2

### GCC/compiler version

11.4.0

### CUDA/cuDNN version

11.8

### GPU model and memory

Nvidia RTX A6000

### Current behavior?

I am trying to build [this custom code ](https://github.com/SamsungLabs/fastflow-tensorflow), but has encountered error when building with GPU support. I have followed the [documentation](https://www.tensorflow.org/install/source) and have set up proper configuration as seen below, setting cuda==Y and confirming the environment path auto detected are indeed valid. 

I thought it is an issue from the custom code but i found out that the same issue persists even in the main repository, r2.7 branch. Moreover, after upgrading bazel to 6.1.0 and build the master branch with same configurations, it was successful. 

I'm convinced this is likely an issue linked to version 2.7.0, though I'm not sure how to tackle this issue. As i don't see a easy way of bypassing this old version since [this repo](https://github.com/SamsungLabs/FastFlow) that i want to try out is dependent on it. 

Any assistance on it is greatly appreciated!

### Standalone code to reproduce the issue

```shell
1. build failure on the custom code producing the output in the log section below

2. build failure on official code r2.7 branch with same configuration and same error

3. upgrade to bazel 6.1.0 and build successfully on official code master branch
```


### Relevant log output

```shell
$ cat .tf_configure.bazelrc
build --action_env PYTHON_BIN_PATH="/usr/bin/python3"
build --action_env PYTHON_LIB_PATH="/usr/lib/python3/dist-packages"
build --python_path="/usr/bin/python3"
build --action_env CUDA_TOOLKIT_PATH="/usr/local/cuda-11.8"
build --action_env TF_CUDA_COMPUTE_CAPABILITIES="8.6"
build --action_env LD_LIBRARY_PATH="/usr/local/cuda-11.8/lib64"
build --action_env GCC_HOST_COMPILER_PATH="/usr/bin/x86_64-linux-gnu-gcc-11"
build --config=cuda
build:opt --copt=-Wno-sign-compare
build:opt --host_copt=-Wno-sign-compare
test --flaky_test_attempts=3
test --test_size_filters=small,medium
test --test_env=LD_LIBRARY_PATH
test:v1 --test_tag_filters=-benchmark-test,-no_oss,-no_gpu,-oss_serial
test:v1 --build_tag_filters=-benchmark-test,-no_oss,-no_gpu
test:v2 --test_tag_filters=-benchmark-test,-no_oss,-no_gpu,-oss_serial,-v1only
test:v2 --build_tag_filters=-benchmark-test,-no_oss,-no_gpu,-v1only

$ bazel build //tensorflow/tools/pip_package:build_pip_package

Starting local Bazel server and connecting to it...
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=98
INFO: Reading rc options for 'build' from /home/haolan/tf-gpu/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /home/haolan/tf-gpu/.bazelrc:
  'build' options: --define framework_shared_object=true --java_toolchain=@tf_toolchains//toolchains/java:tf_java_toolchain --host_java_toolchain=@tf_toolchains//toolchains/java:tf_java_toolchain --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true
INFO: Reading rc options for 'build' from /home/haolan/tf-gpu/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=/usr/bin/python3 --action_env PYTHON_LIB_PATH=/usr/lib/python3/dist-packages --python_path=/usr/bin/python3 --action_env CUDA_TOOLKIT_PATH=/usr/local/cuda-11.8 --action_env TF_CUDA_COMPUTE_CAPABILITIES=8.6 --action_env LD_LIBRARY_PATH=/usr/local/cuda-11.8/lib64: --action_env GCC_HOST_COMPILER_PATH=/usr/bin/x86_64-linux-gnu-gcc-11 --config=cuda
INFO: Reading rc options for 'build' from /home/haolan/tf-gpu/.bazelrc:
  'build' options: --deleted_packages=tensorflow/compiler/mlir/tfrt,tensorflow/compiler/mlir/tfrt/benchmarks,tensorflow/compiler/mlir/tfrt/jit/python_binding,tensorflow/compiler/mlir/tfrt/jit/transforms,tensorflow/compiler/mlir/tfrt/python_tests,tensorflow/compiler/mlir/tfrt/tests,tensorflow/compiler/mlir/tfrt/tests/saved_model,tensorflow/compiler/mlir/tfrt/transforms/lhlo_gpu_to_tfrt_gpu,tensorflow/core/runtime_fallback,tensorflow/core/runtime_fallback/conversion,tensorflow/core/runtime_fallback/kernel,tensorflow/core/runtime_fallback/opdefs,tensorflow/core/runtime_fallback/runtime,tensorflow/core/runtime_fallback/util,tensorflow/core/tfrt/common,tensorflow/core/tfrt/eager,tensorflow/core/tfrt/eager/backends/cpu,tensorflow/core/tfrt/eager/backends/gpu,tensorflow/core/tfrt/eager/core_runtime,tensorflow/core/tfrt/eager/cpp_tests/core_runtime,tensorflow/core/tfrt/fallback,tensorflow/core/tfrt/gpu,tensorflow/core/tfrt/run_handler_thread_pool,tensorflow/core/tfrt/runtime,tensorflow/core/tfrt/saved_model,tensorflow/core/tfrt/saved_model/tests,tensorflow/core/tfrt/tpu,tensorflow/core/tfrt/utils
INFO: Found applicable config definition build:short_logs in file /home/haolan/tf-gpu/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /home/haolan/tf-gpu/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:cuda in file /home/haolan/tf-gpu/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
INFO: Found applicable config definition build:linux in file /home/haolan/tf-gpu/.bazelrc: --copt=-w --host_copt=-w --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++14 --host_cxxopt=-std=c++14 --config=dynamic_kernels --distinct_host_configuration=false --experimental_guard_against_concurrent_changes
INFO: Found applicable config definition build:dynamic_kernels in file /home/haolan/tf-gpu/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
INFO: Repository local_config_cuda instantiated at:
  /home/haolan/tf-gpu/WORKSPACE:15:14: in <toplevel>
  /home/haolan/tf-gpu/tensorflow/workspace2.bzl:1079:19: in workspace
  /home/haolan/tf-gpu/tensorflow/workspace2.bzl:94:19: in _tf_toolchains
Repository rule cuda_configure defined at:
  /home/haolan/tf-gpu/third_party/gpus/cuda_configure.bzl:1448:33: in <toplevel>
ERROR: An error occurred during the fetch of repository 'local_config_cuda':
   Traceback (most recent call last):
	File "/home/haolan/tf-gpu/third_party/gpus/cuda_configure.bzl", line 1401, column 38, in _cuda_autoconf_impl
		_create_local_cuda_repository(repository_ctx)
	File "/home/haolan/tf-gpu/third_party/gpus/cuda_configure.bzl", line 1076, column 27, in _create_local_cuda_repository
		cuda_libs = _find_libs(repository_ctx, check_cuda_libs_script, cuda_config)
	File "/home/haolan/tf-gpu/third_party/gpus/cuda_configure.bzl", line 606, column 21, in _find_libs
		_check_cuda_libs(repository_ctx, check_cuda_libs_script, check_cuda_libs_params.values())
	File "/home/haolan/tf-gpu/third_party/gpus/cuda_configure.bzl", line 501, column 28, in _check_cuda_libs
		checked_paths = execute(repository_ctx, [python_bin, "-c", cmd]).stdout.splitlines()
	File "/home/haolan/tf-gpu/third_party/remote_config/common.bzl", line 230, column 13, in execute
		fail(
Error in fail: Repository command failed
Expected even number of arguments
INFO: Found applicable config definition build:cuda in file /home/haolan/tf-gpu/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
ERROR: @local_config_cuda//:enable_cuda :: Error loading option @local_config_cuda//:enable_cuda: Repository command failed
Expected even number of arguments
```
