

Distributed Backward Pass


Computing dependencies


FAST mode algorithm


SMART mode algorithm




Distributed Optimizer


Simple end to end example


Distributed Optimizer
See the torch.distributed.optim page for documentation on distributed
optimizers.
Design Notes
The distributed autograd design note covers the design of the RPC-
based distributed autograd framework that is useful for applications
such as model parallel training.

Distributed Autograd Design

The RRef design note covers the design of the RRef (Remote REFerence)
protocol used to refer to values on remote workers by the framework.

Remote Reference Protocol

Tutorials
The RPC tutorials introduce users to the RPC framework, provide
several example applications using torch.distributed.rpc APIs, and
demonstrate how to use the profiler to profile RPC-based workloads.

Getting started with Distributed RPC Framework
