Failed to launch ptxas error when using nvidia/cuda runtime Docker image
Hi @goncinious ,

Could you please try installing cuda-nvcc package using `conda install -c "nvidia/label/cuda-11.8.0" cuda-nvcc` and let us know if it helps.

Thanks!
 Thanks @SuryanarayanaY! I'm already using [Poetry](https://python-poetry.org/) for managing dependencies in my use case, so I'd prefer to avoid having to install `conda` just for this. Can I install `cuda-nvcc` without `conda`?
 @goncinious ,

Actually with Docker environment there is no need to install cuda toolkit separately as nvidia-docker takes care of it. But this is applicable to standard tensorflow image not sure to user built images as you are trying to build your own image.Please refer to the official documentation for [docker](https://www.tensorflow.org/install/docker) instructions.

Please try `sudo apt install nvidia-cuda-toolkit` explicitly and let us know if helps. This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. Hi @SuryanarayanaY, thanks for your reply.

> > Actually with Docker environment there is no need to install cuda toolkit separately as nvidia-docker takes care of it. But this is applicable to standard tensorflow image not sure to user built images as you are trying to build your own image.Please refer to the official documentation for [docker](https://www.tensorflow.org/install/docker) instructions.

Yes, I believe the same applies to the `devel` image from the official `nvidia/cuda` image. As mentioned above, it doesn't show the warning highlighted above. However, the point here is how to properly setup `ptxas` compilation while avoiding installing massive dependencies in the `runtime` image.

> Please try sudo apt install nvidia-cuda-toolkit explicitly and let us know if helps.

This indeed solves the issue above, but adds (4.6 GB) to the image, which  the purpose of using the slimmer `runtime` image.


<details><summary>Click to expand!</summary> 

```

 The following NEW packages will be installed:
  adwaita-icon-theme alsa-topology-conf alsa-ucm-conf at-spi2-core ca-certificates-java cpp-8 dbus
  fontconfig fonts-dejavu-extra g++-8 gcc-8 gcc-8-base gtk-update-icon-cache hicolor-icon-theme
  humanity-icon-theme java-common javascript-common libaccinj64-10.1 libapparmor1 libasound2
  libasound2-data libasyncns0 libatk-bridge2.0-0 libatk-wrapper-java libatk-wrapper-java-jni
  libatk1.0-0 libatk1.0-data libatspi2.0-0 libavahi-client3 libavahi-common-data libavahi-common3
  libcairo-gobject2 libcairo2 libcublas10 libcublaslt10 libcudart10.1 libcufft10 libcufftw10
  libcuinj64-10.1 libcups2 libcupti-dev libcupti-doc libcupti10.1 libcurand10 libcusolver10
  libcusolvermg10 libcusparse10 libdatrie1 libdbus-1-3 libdrm-amdgpu1 libdrm-common libdrm-intel1
  libdrm-nouveau2 libdrm-radeon1 libdrm2 libedit2 libegl-dev libegl-mesa0 libegl1 libelf1 libflac8
  libfontenc1 libfribidi0 libgail-common libgail18 libgbm1 libgcc-8-dev libgdk-pixbuf2.0-0
  libgdk-pixbuf2.0-bin libgdk-pixbuf2.0-common libgif7 libgl-dev libgl1 libgl1-mesa-dev
  libgl1-mesa-dri libglapi-mesa libgles-dev libgles1 libgles2 libglvnd-dev libglvnd0 libglx-dev
  libglx-mesa0 libglx0 libgpm2 libgraphite2-3 libgtk2.0-0 libgtk2.0-bin libgtk2.0-common
  libharfbuzz0b libicu66 libjbig0 libjpeg-turbo8 libjpeg8 libjs-jquery libjs-underscore liblcms2-2
  libllvm12 libmpx2 libncurses5 libnppc10 libnppial10 libnppicc10 libnppicom10 libnppidei10
  libnppif10 libnppig10 libnppim10 libnppist10 libnppisu10 libnppitc10 libnpps10 libnspr4 libnss3
  libnvblas10 libnvgraph10 libnvidia-compute-418-server libnvidia-ml-dev libnvjpeg10 libnvrtc10.1
  libnvtoolsext1 libnvvm3 libogg0 libopengl-dev libopengl0 libpango-1.0-0 libpangocairo-1.0-0
  libpangoft2-1.0-0 libpciaccess0 libpcsclite1 libpixman-1-0 libpulse-mainloop-glib0 libpulse0
  librsvg2-2 librsvg2-common libsensors-config libsensors5 libsndfile1 libstdc++-8-dev libthai-data
  libthai0 libthrust-dev libtiff5 libtinfo5 libvdpau-dev libvdpau1 libvorbis0a libvorbisenc2
  libvulkan1 libwayland-client0 libwayland-server0 libwebp6 libwrap0 libx11-xcb1 libxaw7
  libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0 libxcb-present0 libxcb-randr0 libxcb-render0
  libxcb-shape0 libxcb-shm0 libxcb-sync1 libxcb-xfixes0 libxcb-xkb1 libxcomposite1 libxcursor1
  libxdamage1 libxfixes3 libxi6 libxinerama1 libxkbcommon-x11-0 libxkbcommon0 libxkbfile1 libxml2
  libxmu6 libxmuu1 libxpm4 libxrandr2 libxshmfence1 libxtst6 libxv1 libxxf86dga1 libxxf86vm1
  mesa-vdpau-drivers mesa-vulkan-drivers node-html5shiv nsight-compute nsight-systems
  nvidia-cuda-dev nvidia-cuda-doc nvidia-cuda-gdb nvidia-cuda-toolkit nvidia-opencl-dev
  nvidia-profiler nvidia-visual-profiler ocl-icd-libopencl1 ocl-icd-opencl-dev opencl-c-headers
  openjdk-8-jre openjdk-8-jre-headless shared-mime-info ubuntu-mono vdpau-driver-all x11-utils
  xkb-data
0 upgraded, 207 newly installed, 0 to remove and 2 not upgraded.
Need to get 1517 MB of archives.
After this operation, 4628 MB of additional disk space will be used.
```

</details>

Note that this method seems to install a deprecated version of `ptxas`, which we want to avoid too. See below:

> 2023-06-19 11:12:26.647185: E tensorflow/compiler/xla/stream_executor/gpu/asm_compiler.cc:114] *** WARNING *** You are using ptxas 10.1.243, which is older than 11.1. ptxas before 11.1 is known to miscompile XLA code, leading to incorrect results or invalid-address errors.
 Hi @goncinious ,



> Yes, I believe the same applies to the `devel` image from the official `nvidia/cuda` image. As mentioned above, it doesn't show the warning highlighted above. However, the point here is how to properly setup `ptxas` compilation while avoiding installing massive dependencies in the `runtime` image.

AFAIK ptxas is a package that included in CUDA package itself and I am not sure this falls under TF scope.May be we need to confirm the same with Nvidia team. I believe devel images dont have this problem with ptxas path as per reference from the attached issues from Jax, [link1](https://github.com/google/jax/discussions/6843) and [link2](https://github.com/google/jax/discussions/10327).



> 2023-06-19 11:12:26.647185: E tensorflow/compiler/xla/stream_executor/gpu/asm_compiler.cc:114] *** WARNING *** You are using ptxas 10.1.243, which is older than 11.1. ptxas before 11.1 is known to miscompile XLA code, leading to incorrect results or invalid-address errors.

I can see this as logged as Error log may be due to the reason that it might miscompile XLA code.
 @goncinious ,

As per Official TF documentation for Docker support you might need to install Nvidia driver in host system and Nvidia container toolkit on docker, the instructions for which mentioned [here](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#docker).

Could you please check whether the above resources solves your problem

Thanks!

 This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60737">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60737">No</a>
 I followed this guide <https://www.tensorflow.org/install/pip#windows-wsl2> and had the same issue where jupyter notebook kernel was crashing because it couldn't find ptxas. `conda install -c "nvidia/label/cuda-11.8.0" cuda-nvcc` fixed the issue. Thanks.
<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

Yes

### Source

source

### Tensorflow Version

2.14.0-dev20230531

### Custom Code

Yes

### OS Platform and Distribution

20.04

### Mobile device

_No response_

### Python version

3.9.16

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

CUDA 11.8.0 / cuDNN 8.7

### GPU model and memory

_No response_

### Current Behaviour?

I get the following warning when I run prediction using a 3D model (full log provided below):
```
2023-05-31 11:06:53.982167: W tensorflow/compiler/xla/stream_executor/gpu/redzone_allocator.cc:318] INTERNAL: Failed to launch ptxas
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
```
I'm using a model that takes a 3D input and the error occurs when I have a `Conv3DTranspose` layer (which my model implements).

I'm using the official [`nvidia/cuda`](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/cuda) image with runtime libraries, specifically the `nvcr.io/nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu20.0`.

I don't get the warning when using the `devel` image, however, I want to stick with `runtime`, as it's much smaller and I don't need any of the compiling capabilities or dev tools.

Does this warning have an impact on model inference and how can I fix it?

Minimal Dockerfile and Python script to reproduce issue are provided below.

### Standalone code to reproduce the issue

Dockerfile:
```shell

FROM nvcr.io/nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu20.04

ENV PYENV_ROOT=/root/.pyenv \
    PATH="/root/.local/bin:/opt/venv/bin:/root/.pyenv/shims/:root/.pyenv/bin:${PATH}" \
    DEBIAN_FRONTEND=noninteractive

ARG PYTHON_VERSION=3.9

RUN apt update -y && apt upgrade -y && \
    apt-get install --no-install-recommends -y wget build-essential curl git && \
    # install python via pyenv
    apt-get install -y --no-install-recommends libreadline-gplv2-dev libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev libffi-dev zlib1g-dev liblzma-dev && \
    curl https://pyenv.run | bash && \
    pyenv update && \
    pyenv install ${PYTHON_VERSION} && \
    pyenv global ${PYTHON_VERSION} && \
    pyenv rehash && \
    # install tensorflow
    python${PYTHON_MAJOR_VERSION} -m pip install tf-nightly
```

`test.py`:
```python
import numpy as np
import tensorflow as tf
from tensorflow.keras import Input
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Conv3DTranspose

def build_model(input_shape):
    inputs = Input(shape=input_shape)
    x = Conv3DTranspose(filters=1, kernel_size=(1, 1, 1), strides=(1, 1, 1))(inputs)
    return Model(inputs=inputs, outputs=x)

print(tf.version.GIT_VERSION, tf.version.VERSION)

input_shape = (10, 10, 10, 1)
model = build_model(input_shape)

# input of shape (5, 10, 10, 10, 1)
test_input = np.ones(shape=(5, *input_shape), dtype=np.float32)

output = model.predict(test_input, verbose=1, batch_size=1)
```

Steps to reproduce:
```
docker build -t tf-issue -f /path/to/Dockerfile .
docker run --gpus all -v /path/to/test.py:/test.py tf-issue python /test.py
```


### Relevant log output

```shell
==========
== CUDA ==
==========

CUDA Version 11.8.0

Container image Copyright (c) 2016-2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

2023-05-31 11:06:49.787277: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:7704] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-05-31 11:06:49.787340: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-05-31 11:06:49.787352: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1520] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-05-31 11:06:49.794051: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-05-31 11:06:50.626364: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-05-31 11:06:51.483784: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-31 11:06:51.504146: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-31 11:06:51.504472: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-31 11:06:51.505671: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-31 11:06:51.505938: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-31 11:06:51.506189: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-31 11:06:52.004673: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-31 11:06:52.004929: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-31 11:06:52.005143: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-05-31 11:06:52.005338: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 20638 MB memory:  -> device: 0, name: NVIDIA A10G, pci bus id: 0000:00:1e.0, compute capability: 8.6
2023-05-31 11:06:53.285968: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:434] Loaded cuDNN version 8700
2023-05-31 11:06:53.981061: I tensorflow/tsl/platform/default/subprocess.cc:304] Start cannot spawn child process: No such file or directory
2023-05-31 11:06:53.981512: I tensorflow/tsl/platform/default/subprocess.cc:304] Start cannot spawn child process: No such file or directory
2023-05-31 11:06:53.981543: W tensorflow/compiler/xla/stream_executor/gpu/asm_compiler.cc:109] Couldn't get ptxas version : FAILED_PRECONDITION: Couldn't get ptxas/nvlink version string: INTERNAL: Couldn't invoke ptxas --version
2023-05-31 11:06:53.982117: I tensorflow/tsl/platform/default/subprocess.cc:304] Start cannot spawn child process: No such file or directory
2023-05-31 11:06:53.982167: W tensorflow/compiler/xla/stream_executor/gpu/redzone_allocator.cc:318] INTERNAL: Failed to launch ptxas
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
v1.12.1-94762-gf8066222ad6 2.14.0-dev20230531
5/5 [==============================] - 2s 1ms/step
```
</details>