Quickstart
To launch a fault-tolerant job, run the following on all nodes.
torchrun
      --nnodes=NUM_NODES
      --nproc_per_node=TRAINERS_PER_NODE
      --max_restarts=NUM_ALLOWED_FAILURES
      --rdzv_id=JOB_ID
      --rdzv_backend=c10d
      --rdzv_endpoint=HOST_NODE_ADDR
      YOUR_TRAINING_SCRIPT.py (--arg1 ... train script args...)
To launch an elastic job, run the following on at least "MIN_SIZE"
nodes and at most "MAX_SIZE" nodes.
torchrun
       --nnodes=MIN_SIZE:MAX_SIZE
       --nproc_per_node=TRAINERS_PER_NODE
       --max_restarts=NUM_ALLOWED_FAILURES_OR_MEMBERSHIP_CHANGES
       --rdzv_id=JOB_ID
       --rdzv_backend=c10d
       --rdzv_endpoint=HOST_NODE_ADDR
       YOUR_TRAINING_SCRIPT.py (--arg1 ... train script args...)
Note:
TorchElastic models failures as membership changes. When a node
  fails, this is treated as a "scale down" event. When the failed node
  is replaced by the scheduler, it is a "scale up" event. Hence for