Multiprocessing
Library that launches and manages "n" copies of worker subprocesses
either specified by a function or a binary.
For functions, it uses "torch.multiprocessing" (and therefore python
"multiprocessing") to spawn/fork worker processes. For binaries it
uses python "subprocessing.Popen" to create worker processes.
Usage 1: Launching two trainers as a function
from torch.distributed.elastic.multiprocessing import Std, start_processes
def trainer(a, b, c):
       pass # train
# runs two trainers
   # LOCAL_RANK=0 trainer(1,2,3)
   # LOCAL_RANK=1 trainer(4,5,6)
   ctx = start_processes(
           name="trainer",
           entrypoint=trainer,
           args={0: (1,2,3), 1: (4,5,6)},
           envs={0: {"LOCAL_RANK": 0}, 1: {"LOCAL_RANK": 1}},
           log_dir="/tmp/foobar",
           redirects=Std.ALL, # write all worker stdout/stderr to a log file
           tee={0: Std.ERR}, # tee only local rank 0's stderr to console
         )