How to get outputs in the same order as inputs with multiple spawned processes running on multiple GPUs and batches of data processed by each?
I am using Pytorch Distributed Data Parallel approach and spawning multiple processes, each running on separate GPU.I am using Pytorch Distributed Data Sampler along with Data Loader for loading batches of input data to each process.
My questions:

Under the hood, how does Pytorch Distributed Data Sampler, Data Loader make slices of input data? Just for simplicity say we have 4 GPUs, and 400 input samples and batch size of say 50, then will Pytorch Distributed Data Sampler (together with Data Loader) make first 50 samples go to GPU-0, next 50 to GPU-1., next 50 to GPU-2, then GPU-3 and then again next 50 to GPU-0 i.e. in the order of GPU device number? or the order of GPU to select for next batch of input is random based on which GPU has finished its previous batch first? or is it like 400 samples get divided into 4 parts first and then GPU-0 would get first 100 samples of input data (50 at a time ),  GPU-1 will get next 100 samples ( 50 at a time) and so on..and in this case no matter if say GPU-3 gets its second batch started earlier than GPU-0, but still with respect to input data, GPU-0 would still have first 100 samples and GPU-3 would have last 100?

2). My Second question is how to retrieve output data in same order as input data so that final consolidated output ( having outputs from all processes combined in one data structure) is in same order as original inputs and each output corresponds to the right input


The PyTorch documentation on DistributedSampler doesn't provide any guarantees regarding how data is distributed across processes and devices, other than the fact that it is, in fact, distributed across processes and devices. You shouldn't design your application to be dependent on an implementation detail of an external package; otherwise, your application could suddenly fail one day after updating PyTorch, and you'd have no idea why (or potentially that it's even failing to begin with). If, for some reason, you absolutely need the data to be distributed in a very specific way, you should roll your own solution. The documentation for DistributedDataParallel suggests that, if you're using a single host with N GPUs, you should spin up N processes, each designated a single GPU. A simple solution would be to set the process's rank equal to the designated GPU device ID; this could in-turn be used in a custom sampler class to select the appropriate sample indices.
You could try to control the order in which outputs are returned by the various distributed processes, but this introduces unnecessary synchronization which would defeat much of the purpose of parallelization. A better solution is to simply return outputs in an arbitrary order, and then sort them after-the-fact. If you'd like the outputs to be sorted in the same order as the inputs, you can just associate each input with, say, an integer index (input 0 gets index 0, input 1 gets index 1, and so on). When returning the output, also return the index of the associated input (e.g. as a tuple). Afterwards, you can just sort the outputs by their corresponding indices.

