Support CUDA Compute 9.0 for NVIDIA H100
Seconding this request.  We have compute 8.9 GPUs now (L40) and it would be nice to get cuda12 support in tensorflow from pypi rather than having to build from source. So I get that Tensorflow native installation does not work with H100 which I can also reproduce on our HPC, but is there a workaround for that? Does building it from source with Bazel work for anyone? Also interested to know if any one has been able to build from source with Compute Capability 9.0? my first try failed so far. @kapoorlab installing tf-nightly worked for me and able to detect gpus. @dwyatte , Could you please check with `tf-nightly` and let us know the outcome. Thanks! @purvang3 were you able to fully run an actual code without getting any compilation and runtime errors as just being able to detect GPUs does not mean tf would run on the hardware with such CUDA capability requirement. I tested my code with TF 2.13.1 and TF 2.15.0_rc0 on a Machine with 8xH100. I can train a model on the machine using all 8 GPUs. But the starting process takes really long. 
And I do get the warning "TensorFlow was not built with CUDA kernel binaries compatible with compute capability 9.0. CUDA kernels will be jit-compiled from PTX, which could take 30 minutes or longer."
Is it then using the full potential of the GPUs? Hi @dwyatte , does the commit here solves your issue https://github.com/tensorflow/tensorflow/commit/c71bcf6fe2cf45eddec5379c083e99b8d611337b Hi @sachinprasadhs, I'm having trouble accessing H100s from my cloud provider at the moment so can't confirm CUDA Compute 9.0 support -- if anyone else can do that and comment here, that would be great, otherwise I'll confirm once I can access an H100

I can however confirm that TensorFlow 2.15 supports CUDA Compute 8.9 and works with L4* (may be of interest to @Lana-Deere) I have been able to make it work on H100 following these instructions:


```console
conda create -n tensorflowGPU python=3.10
conda install mamba -c conda-forge
mamba install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
mamba install -c nvidia cuda-nvcc=11.3.58
python3 -m pip install nvidia-cudnn-cu11==8.6.0.163 tensorflow-gpu==2.10.*
mkdir -p $CONDA_PREFIX/etc/conda/activate.d
echo 'CUDNN_PATH=$(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
echo 'export LD_LIBRARY_PATH=$CONDA_PREFIX/lib/:$CUDNN_PATH/lib:$LD_LIBRARY_PATH' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
source $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
# Verify install:
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

```

The last command should show a message like (tested on H100):

```
2023-07-20 14:52:48.833385: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-07-20 14:52:51.703785: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2048] TensorFlow was not built with CUDA kernel binaries compatible with compute capability 9.0. CUDA kernels will be jit-compiled from PTX, which could take 30 minutes or longer.
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

```

```

spack load cuda@11.8.0
spack load cudnn@8.6.0.163-11.8
mkdir -p $CONDA_PREFIX/lib/nvvm/libdevice/
cp -p $CONDA_PREFIX/lib/libdevice.10.bc $CONDA_PREFIX/lib/nvvm/libdevice/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/
export XLA_FLAGS=--xla_gpu_cuda_data_dir=$CONDA_PREFIX 
```
Initially I had to load specific cuda and cudnn library versions but I found it to be quite flexible in that regard. So the issue can be closed. > I have been able to make it work on H100 following these instructions:
> 
> 
> ```console
> conda create -n tensorflowGPU python=3.10
> conda install mamba -c conda-forge
> mamba install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0
> mamba install -c nvidia cuda-nvcc=11.3.58
> python3 -m pip install nvidia-cudnn-cu11==8.6.0.163 tensorflow-gpu==2.10.*
> mkdir -p $CONDA_PREFIX/etc/conda/activate.d
> echo 'CUDNN_PATH=$(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
> echo 'export LD_LIBRARY_PATH=$CONDA_PREFIX/lib/:$CUDNN_PATH/lib:$LD_LIBRARY_PATH' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
> source $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
> # Verify install:
> python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
> 
> ```
> 
> The last command should show a message like (tested on H100):
> 
> ```
> 2023-07-20 14:52:48.833385: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
> 2023-07-20 14:52:51.703785: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2048] TensorFlow was not built with CUDA kernel binaries compatible with compute capability 9.0. CUDA kernels will be jit-compiled from PTX, which could take 30 minutes or longer.
> [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
> 
> ```
> 
> ```
> 
> spack load cuda@11.8.0
> spack load cudnn@8.6.0.163-11.8
> mkdir -p $CONDA_PREFIX/lib/nvvm/libdevice/
> cp -p $CONDA_PREFIX/lib/libdevice.10.bc $CONDA_PREFIX/lib/nvvm/libdevice/
> export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/
> export XLA_FLAGS=--xla_gpu_cuda_data_dir=$CONDA_PREFIX 
> ```
> Initially I had to load specific cuda and cudnn library versions but I found it to be quite flexible in that regard. So the issue can be closed.

Why is it still getting Jit compiled? This should not be happening if it was build with compute 9 capabilities.  
> Why is it still getting Jit compiled? This should not be happening if it was build with compute 9 capabilities.

I had it working before the build with compute 9 capabilities was made available, have not tried it with the commit that claims to solve that bit. Can confirm this is fixed on nightly. Here are the versions I used

tf-nightly==2.17.0.dev20240308
tf_keras-nightly==2.17.0.dev2024030810
tensorflow-text-nightly==2.17.0.dev20240306

I'm guessing the fix went in awhile back but this should be enough to close this issue Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60739">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60739">No</a>
 > tf-nightly==2.17.0.dev20240308

how to fix this using python 3.8 which `tf-nightly` does not support? 
<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Feature Request

### Have you reproduced the bug with TF nightly?

Yes

### Source

binary

### Tensorflow Version

v1.12.1-94762-gf8066222ad6 2.14.0-dev20230531

### Custom Code

Yes

### OS Platform and Distribution

_No response_

### Mobile device

_No response_

### Python version

_No response_

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current Behaviour?

With NVIDIA H100 general availability on the horizon, it would be nice to start supporting CUDA compute capability 9.0. Are there plans for this in an upcoming release or could it be added to nightly builds?

In testing my code on an H100, I currently see 

```
2023-05-31 14:45:08.138842: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2052] TensorFlow was not built with CUDA kernel binaries compatible with compute capability 9.0. CUDA kernels will be jit-compiled from PTX, which could take 30 minutes or longer.
```

and after jit-compiling, my code that works fine on NVIDIA A100 GPUs eventually fails during allocation. I'd like to rule out that it's not a problem with jit-compiling, so having CUDA compute capability 9.0 compiled into tf-nightly would be one way to rule that out.

### Standalone code to reproduce the issue

```shell
N/A
```


### Relevant log output

_No response_</details>