How does one run PyTorch on a A40 GPU without errors (with DDP too)?
I tried running my pytorch code but got this error:
A40 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_61 sm_70 sm_75 compute_37.
If you want to use the A40 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(incompatible_device_warn.format(device_name, capability, &quot; &quot;.join(arch_list), device_name))
Using backend: pytorch
/home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/lib/python3.8/site-packages/torch/cuda/__init__.py:104: UserWarning: 
A40 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_61 sm_70 sm_75 compute_37.
If you want to use the A40 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(incompatible_device_warn.format(device_name, capability, &quot; &quot;.join(arch_list), device_name))
Traceback (most recent call last):
  File &quot;/home/miranda9/ML4Coq/ml4coq-proj-src/embeddings_zoo/tree_nns/main_brando.py&quot;, line 305, in &lt;module&gt;
    main_distributed()
  File &quot;/home/miranda9/ML4Coq/ml4coq-proj-src/embeddings_zoo/tree_nns/main_brando.py&quot;, line 201, in main_distributed
    mp.spawn(fn=train, args=(opts,), nprocs=opts.world_size)
  File &quot;/home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/lib/python3.8/site-packages/torch/multiprocessing/spawn.py&quot;, line 199, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File &quot;/home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/lib/python3.8/site-packages/torch/multiprocessing/spawn.py&quot;, line 157, in start_processes
    while not context.join():
  File &quot;/home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/lib/python3.8/site-packages/torch/multiprocessing/spawn.py&quot;, line 118, in join
    raise Exception(msg)
Exception: 

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File &quot;/home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/lib/python3.8/site-packages/torch/multiprocessing/spawn.py&quot;, line 19, in _wrap
    fn(i, *args)
  File &quot;/home/miranda9/ML4Coq/ml4coq-proj-src/embeddings_zoo/tree_nns/main_brando.py&quot;, line 210, in train
    setup_process(opts, rank, master_port=opts.master_port, world_size=opts.world_size)
  File &quot;/home/miranda9/ultimate-utils/ultimate-utils-proj-src/uutils/torch/distributed.py&quot;, line 165, in setup_process
    dist.init_process_group(backend, rank=rank, world_size=world_size)
  File &quot;/home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py&quot;, line 455, in init_process_group
    barrier()
  File &quot;/home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py&quot;, line 1960, in barrier
    work = _default_pg.barrier()
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1607369981906/work/torch/lib/c10d/ProcessGroupNCCL.cpp:31, unhandled cuda error, NCCL version 2.7.8


but then it sends me to download it for my mac...? which is weird. What version of
pytorch, cuda, cudnn, nccl and other things do I need for a GPU A40?
to see the code I ran and conda env info see this: https://github.com/pytorch/pytorch/issues/58794

related links

https://github.com/pytorch/pytorch/issues/45021
https://github.com/pytorch/pytorch/issues/45028
https://github.com/pytorch/pytorch/issues/58794
How does one install pytorch 1.9 in an HPC that seems to refuse to cooperate?


My guess is the following:
A40 gpus have CUDA capability of sm_86 and they are only compatible with CUDA &gt;= 11.0. But CUDA &gt;= 11.0 is only compatible with PyTorch &gt;= 1.7.0 I believe.
So do:
conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=11.0 -c pytorch

or
conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch

or
conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch

if you are in an HPC you might want to do:
module load gcc/9.2.0

#module load cuda-toolkit/10.2
module load cuda-toolkit/11.1


this seemed to work:
(metalearning) miranda9~/automl-meta-learning $ python -c &quot;import uutils; uutils.torch_uu.gpu_test()&quot;


device name: A40

Success, no Cuda errors means it worked see:
out=tensor([[2.3272],
        [5.6796]], device='cuda:0')
(metalearning) miranda9~/automl-meta-learning $ 
(metalearning) miranda9~/automl-meta-learning $ 
(metalearning) miranda9~/automl-meta-learning $ 
(metalearning) miranda9~/automl-meta-learning $ conda list | grep torch
_pytorch_select           0.1                       cpu_0  
pytorch                   1.7.1           py3.9_cuda11.0.221_cudnn8.0.5_0    pytorch
torchaudio                0.7.2                      py39    pytorch
torchmeta                 1.7.0                    pypi_0    pypi
torchvision               0.8.2           cpu_py39ha229d99_0 

