Running a PyTorch dataloader/Dataset on multiple distributed CPUs
I wonder if there is a way to distributed the dataloader/Dataset to many CPUs, even when using a single GPU.
Specifically, I would like to have a Dataset class, and the __getitem__ function will be distributed across many different CPUs (using mpi maybe? but any other way is also good).
Thanks
EDIT
My title was erroneously edited, I am not trying to distribute the model itslef, I only want to distribute the data loading/parsing of the model
EDIT - 2
Some interesting discussion in this direction is available here

You can do this of course, but mind you - it is not always very effective for general Machine Learning needs, due to the hefty communication costs.
Use DistributedDataParallel


  Implements distributed data parallelism that is based on
  torch.distributed package at the module level.
  
  This container parallelizes the application of the given module by
  splitting the input across the specified devices by chunking in the
  batch dimension. The module is replicated on each machine and each
  device, and each such replica handles a portion of the input. During
  the backwards pass, gradients from each node are averaged.


In practice, I'd recommend you utilize the pytorch_lightning package, to reduce some of the boilerplate code you need to write for this to work. 

Reference: 
DistributedDataParallel,pytorch_lightning
