How can I concatenate pytorch tensors or lists in a distributed multi-node setup?
I am trying to implement something like this for 2 nodes (each node with 2 GPUs):
#### Parallel process initiated with torch.distributed.init_process_group()
### All GPUs work in parallel, and generate lists like :
    [20, 0, 1, 17] for GPU0 of node A 
    [1, 2, 3, 4] for GPU1 of node A
    [5, 6, 7, 8] for GPU0 of node B
    [0, 2, 4, 6] for GPU1 of node B

I tried
torch.distributed.reduce()

to get a sum of these 4:
[26,  10, 15, 35]

But what I want is a concatenated version like this
[[20, 0, 1, 17], [1, 2, 3, 4] , [5, 6, 7, 8] , [0, 2, 4, 6]]

Or
[20, 0, 1, 17, 1, 2, 3, 4, 5, 6, 7, 8, 0, 2, 4, 6]

is also OK with me.
Is it possible to achieve this from torch.distributed?

You can use dist.all_gather to do this:
import torch
import torch.distributed as dist

q = torch.tensor([20, 0, 1, 17]) # generated on each gpu (with different values) as you mentioned
all_q = [torch.zeros_like(q) for _ in range(world_size)] # world_size is the total number of gpu processes you are running. 4 in your case.
all_q = dist.all_gather(all_q, q)

all_q would then have the following:
[torch.tensor([20, 0, 1, 17]), torch.tensor([1, 2, 3, 4]), torch.tensor([5, 6, 7, 8]), torch.tensor([0, 2, 4, 6])]
You can then use torch.cat to collapse all elements into one array if you like.
You can use dist.all_gather_multigpu if you list of lists of tensors.
