title: How does one use Pytorch (+ cuda) with an A100 GPU?
body: <p>I was trying to use my current code with an A100 gpu but I get this error:</p>
  <pre><code>---&gt; backend='nccl'
  /home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/lib/python3.8/site-packages/torch/cuda/__init__.py:104: UserWarning: 
  A100-SXM4-40GB with CUDA capability sm_80 is not compatible with the current PyTorch installation.
  The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_61 sm_70 sm_75 compute_37.
  If you want to use the A100-SXM4-40GB GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
  </code></pre>
  <p>which is reather confusing because it points to the usual pytorch installation but doesn't tell me which combination of pytorch version + cuda version to use for my specific hardware (A100). What is the right way to install pytorch for an A100?</p>
  <hr />
  <p>These are some versions I've tried:</p>
  <pre><code># conda install -y pytorch==1.8.0 torchvision cudatoolkit=10.2 -c pytorch
  # conda install -y pytorch torchvision cudatoolkit=10.2 -c pytorch
  #conda install -y pytorch==1.7.1 torchvision torchaudio cudatoolkit=10.2 -c pytorch -c conda-forge
  # conda install -y pytorch==1.6.0 torchvision cudatoolkit=10.2 -c pytorch
  #conda install -y pytorch==1.7.1 torchvision torchaudio cudatoolkit=11.1 -c pytorch -c conda-forge
  
  # conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch
  # conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c conda-forge
  # conda install -y pytorch torchvision cudatoolkit=9.2 -c pytorch # For Nano, CC
  # conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c conda-forge
  </code></pre>
  <hr />
  <p>note that this can be subtle because I've had this error with this machine + pytorch version in the past:</p>
  <p><a href="https://stackoverflow.com/questions/66807131/how-to-solve-the-famous-unhandled-cuda-error-nccl-version-2-7-8-error">How to solve the famous `unhandled cuda error, NCCL version 2.7.8` error?</a></p>
  <hr />
  <h1>Bonus 1:</h1>
  <p>I still have errors:</p>
  <pre><code>ncclSystemError: System call (socket, malloc, munmap, etc) failed.
  Traceback (most recent call last):
    File &quot;/home/miranda9/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_dist_maml_l2l.py&quot;, line 1423, in &lt;module&gt;
      main()
    File &quot;/home/miranda9/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_dist_maml_l2l.py&quot;, line 1365, in main
      train(args=args)
    File &quot;/home/miranda9/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_dist_maml_l2l.py&quot;, line 1385, in train
      args.opt = move_opt_to_cherry_opt_and_sync_params(args) if is_running_parallel(args.rank) else args.opt
    File &quot;/home/miranda9/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/distributed.py&quot;, line 456, in move_opt_to_cherry_opt_and_sync_params
      args.opt = cherry.optim.Distributed(args.model.parameters(), opt=args.opt, sync=syn)
    File &quot;/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/cherry/optim.py&quot;, line 62, in __init__
      self.sync_parameters()
    File &quot;/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/cherry/optim.py&quot;, line 78, in sync_parameters
      dist.broadcast(p.data, src=root)
    File &quot;/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py&quot;, line 1090, in broadcast
      work = default_pg.broadcast([tensor], opts)
  RuntimeError: NCCL error in: ../torch/lib/c10d/ProcessGroupNCCL.cpp:911, unhandled system error, NCCL version 2.7.8
  </code></pre>
  <p>one of the answers suggested to have nvcca &amp; pytorch.version.cuda to match but they do not:</p>
  <pre><code>(meta_learning_a100) [miranda9@hal-dgx ~]$ python -c &quot;import torch;print(torch.version.cuda)&quot;
  
  11.1
  (meta_learning_a100) [miranda9@hal-dgx ~]$ nvcc -V
  nvcc: NVIDIA (R) Cuda compiler driver
  Copyright (c) 2005-2020 NVIDIA Corporation
  Built on Wed_Jul_22_19:09:09_PDT_2020
  Cuda compilation tools, release 11.0, V11.0.221
  Build cuda_11.0_bu.TC445_37.28845127_0
  </code></pre>
  <p>How do I match them? I this the error? Can someone display their pip, conda and nvcca version to see what set up works?</p>
  <p>More error messages:</p>
  <pre><code>hal-dgx:21797:21797 [0] NCCL INFO Bootstrap : Using [0]enp226s0:141.142.153.83&lt;0&gt; [1]virbr0:192.168.122.1&lt;0&gt;
  hal-dgx:21797:21797 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
  hal-dgx:21797:21797 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB ; OOB enp226s0:141.142.153.83&lt;0&gt;
  hal-dgx:21797:21797 [0] NCCL INFO Using network IB
  NCCL version 2.7.8+cuda11.1
  hal-dgx:21805:21805 [2] NCCL INFO Bootstrap : Using [0]enp226s0:141.142.153.83&lt;0&gt; [1]virbr0:192.168.122.1&lt;0&gt;
  hal-dgx:21799:21799 [1] NCCL INFO Bootstrap : Using [0]enp226s0:141.142.153.83&lt;0&gt; [1]virbr0:192.168.122.1&lt;0&gt;
  hal-dgx:21805:21805 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
  hal-dgx:21799:21799 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
  hal-dgx:21811:21811 [3] NCCL INFO Bootstrap : Using [0]enp226s0:141.142.153.83&lt;0&gt; [1]virbr0:192.168.122.1&lt;0&gt;
  hal-dgx:21811:21811 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
  hal-dgx:21811:21811 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB ; OOB enp226s0:141.142.153.83&lt;0&gt;
  hal-dgx:21811:21811 [3] NCCL INFO Using network IB
  hal-dgx:21799:21799 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB ; OOB enp226s0:141.142.153.83&lt;0&gt;
  hal-dgx:21805:21805 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB ; OOB enp226s0:141.142.153.83&lt;0&gt;
  hal-dgx:21799:21799 [1] NCCL INFO Using network IB
  hal-dgx:21805:21805 [2] NCCL INFO Using network IB
  
  hal-dgx:21797:27906 [0] misc/ibvwrap.cc:280 NCCL WARN Call to ibv_create_qp failed
  hal-dgx:21797:27906 [0] NCCL INFO transport/net_ib.cc:360 -&gt; 2
  hal-dgx:21797:27906 [0] NCCL INFO transport/net_ib.cc:437 -&gt; 2
  hal-dgx:21797:27906 [0] NCCL INFO include/net.h:21 -&gt; 2
  hal-dgx:21797:27906 [0] NCCL INFO include/net.h:51 -&gt; 2
  hal-dgx:21797:27906 [0] NCCL INFO init.cc:300 -&gt; 2
  hal-dgx:21797:27906 [0] NCCL INFO init.cc:566 -&gt; 2
  hal-dgx:21797:27906 [0] NCCL INFO init.cc:840 -&gt; 2
  hal-dgx:21797:27906 [0] NCCL INFO group.cc:73 -&gt; 2 [Async thread]
  
  hal-dgx:21811:27929 [3] misc/ibvwrap.cc:280 NCCL WARN Call to ibv_create_qp failed
  hal-dgx:21811:27929 [3] NCCL INFO transport/net_ib.cc:360 -&gt; 2
  hal-dgx:21811:27929 [3] NCCL INFO transport/net_ib.cc:437 -&gt; 2
  hal-dgx:21811:27929 [3] NCCL INFO include/net.h:21 -&gt; 2
  hal-dgx:21811:27929 [3] NCCL INFO include/net.h:51 -&gt; 2
  hal-dgx:21811:27929 [3] NCCL INFO init.cc:300 -&gt; 2
  hal-dgx:21811:27929 [3] NCCL INFO init.cc:566 -&gt; 2
  hal-dgx:21811:27929 [3] NCCL INFO init.cc:840 -&gt; 2
  hal-dgx:21811:27929 [3] NCCL INFO group.cc:73 -&gt; 2 [Async thread]
  </code></pre>
  <p>after putting</p>
  <pre><code>import os
  os.environ[&quot;NCCL_DEBUG&quot;] = &quot;INFO&quot;
  </code></pre>