title: How do I specify what version of CUDA libraries my code should be looking for in this container?
body: <p>So my workflow is gonna be a bit wonky, but I'm not permitted to use Docker, so I have Singularity instead.</p>
  <p>I'm running some code that is giving me this error:</p>
  <pre><code>RuntimeError: nvrtc: error: failed to open libnvrtc-builtins.so.11.1.
    Make sure that libnvrtc-builtins.so.11.1 is installed correctly.
  nvrtc compilation failed:
  </code></pre>
  <p>If more details are needed, I can provide.</p>
  <p>I'm operating in a Singularity container built from a docker_hub image I created here: <a href="https://hub.docker.com/repository/docker/elavizadeh/imaginaire_cuda_10.2/builds" rel="nofollow noreferrer">https://hub.docker.com/repository/docker/elavizadeh/imaginaire_cuda_10.2/builds</a>
  (Ignore the fact that the title says cuda_10.2, I'll fix this all up if and when this works.)</p>
  <p>Which is based on this dockerfile I created here: <a href="https://github.com/EvanLavizadeh/imaginaire-1/blob/master/Dockerfile" rel="nofollow noreferrer">https://github.com/EvanLavizadeh/imaginaire-1/blob/master/Dockerfile</a></p>
  <p>In that Dockerfile, I specify a base image &quot;FROM nvcr.io/nvidia/pytorch:20.12-py3&quot;
  <a href="https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel_20-12.html#rel_20-12" rel="nofollow noreferrer">https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel_20-12.html#rel_20-12</a></p>
  <p>I specifically chose that version of that base image because it claims it has Cuda toolkit 11.1.1 which I am hoping will contain <code>libnvrtc-builtins.so.11.1</code>.</p>
  <p>But when I spin up my container, cuda reports version 11.0.221:</p>
  <pre><code>Singularity&gt; nvcc -V
  nvcc: NVIDIA (R) Cuda compiler driver
  Copyright (c) 2005-2020 NVIDIA Corporation
  Built on Wed_Jul_22_19:09:09_PDT_2020
  Cuda compilation tools, release 11.0, V11.0.221
  Build cuda_11.0_bu.TC445_37.28845127_0
  </code></pre>
  <p>Additionally, when I look at the files in <code>/usr/local/cuda/lib64</code>, I see <code>libnvrtc-builtins.so.11.0</code> but not <code>libnvrtc-builtins.so.11.1</code></p>
  <p>I believe this is the source of my error but I don't know how to correct it. I either need my code to use <code>libnvrtc-builtins.so.11.0</code> instead of <code>libnvrtc-builtins.so.11.1</code> or I need my container to build Cuda 11.1 and hopefully generate <code>libnvrtc-builtins.so.11.0</code>.</p>