How to use a custom Rank to GPU mapping in Pytorch Distributed
I'm running PyTorch distributed on a single 4-gpu node, where I am not the only one running projects. As a result, sometimes certain GPUs are too busy, while others are idle.
Let's say GPU 0 and GPU 2 are busy, but GPU 1 and GPU 3 are not, so I want to use just these GPUs. It seems that by default, Rank 0 is always GPU 0 in pytorch distributed.
How can I specify to PyTorch that Rank 0 should correspond to GPU 1, and Rank 1 should correspond to GPU 3?

You can tell any command which GPUs should be visible to them by setting the variable CUDA_VISIBLE_DEVICES. E.g.
CUDA_VISIBLE_DEVICES=2,3 python run.py &lt;args&gt;

This will only make GPU 2 and 3 visible to that process and thus DDP will only run on those GPUs that you specified. (Note that this is zero-indexed, i.e. 0 is the first GPU.)
