How does DataParallel figure out which gpu I want to use?
I want to find a simple way to specify the gpus that my experiments run on. Currently, I know I can use prepend my python command with CUDA_VISIBLE_DEVICES=1,2,3,4 to set the gpu, and I am guessing DataParallel will then try to use all the gpu.

Is there a way to tell DataParallel directly the ids, like 4,7,9,12?

Yeah, DataParallem provides us the feature of directly passing the gpu ids. 

As per the official documentation here, Data Parallelism is implemented using torch.nn.DataParallel. One can wrap a Module in DataParallel and it will be parallelized over multiple GPUs in the batch dimension.

torch.nn.DataParallel(module, device_ids=None, output_device=None, dim=0)


In your case, you can simply do something like this:

torch.nn.DataParallel(model, device_ids=[4, 7, 9, 12])
output = net(input_var)  # input_var can be on any device, including CPU


You can know more about how to pass gpu ids directly to DataParallel in below links:


MULTI-GPU EXAMPLES  
DataParallel layers

