Get Training error if setting CUDA VISIBLE DEVICE
I am using PyTorch and Cuda 10.1.  If I set CUDA VISIBLE DEVICE in training, the loss is always NAN, if I don't set CUDA VISIBLE DEVICE, everything is working well. Does anyone know what's the problem is?

CUDA_VISIBLE_DEVICES is an os level variable stored in CUDA files I believe. It controls which of your machine's GPUs are made available to perform CUDA computations. It must be set prior to running your code.

If you are trying to control if pytorch uses GPUs and which ones, you should use the built-in pytorch.cuda package for device management.

 import torch

 n_gpus = torch.cuda.device_count()

 if n_gpus &gt; 0:
      device = torch.device("cuda:0") # first device as indexed by pytorch cuda
      print("cuda:0 is device {}".format(torch.cuda.get_device_name(device))) # prints name of device

 if n_gpus &gt; 1:  # if you have more than one device, and so on
      device2 = torch.device("cuda:1")
      print("cuda:1 is device {}".format(torch.cuda.get_device_name(device2)))

 # from here, decide which device you want to use and
 # transfer files to this device accordingly
 model.to(device)
 x.to(device2)
 # etc.


The only reason why you'd want to use CUDA_VISIBLE_DEVICES is if you have multiple GPUs and you need some of them to be available for Cuda / Pytorch tasks, and other GPUs to be available for non-cuda tasks, and are worried about the small amount of GPU memory that torch.cuda packages consume on the GPU when registered as pytorch devices. For most applications this isn't necessary and you should just use pytorch's device management.
