Distributed RPC Framework
The distributed RPC framework provides mechanisms for multi-machine
model training through a set of primitives to allow for remote
communication, and a higher-level API to automatically differentiate
models split across several machines.
Warning:
APIs in the RPC package are stable. There are multiple ongoing work
  items to improve performance and error handling, which will ship in
  future releases.
Warning:
CUDA support was introduced in PyTorch 1.9 and is still a beta
  feature. Not all features of the RPC package are yet compatible with
  CUDA support and thus their use is discouraged. These unsupported
  features include: RRefs, JIT compatibility, dist autograd and dist
  optimizer, and profiling. These shortcomings will be addressed in
  future releases.
Note:
Please refer to PyTorch Distributed Overview for a brief
  introduction to all features related to distributed training.
Basics