both fault tolerant and elastic jobs, "--max_restarts" is used to
  control the total number of restarts before giving up, regardless of
  whether the restart was caused due to a failure or a scaling event.
"HOST_NODE_ADDR", in form [:] (e.g.
node1.example.com:29400), specifies the node and the port on which the
C10d rendezvous backend should be instantiated and hosted. It can be
any node in your training cluster, but ideally you should pick a node
that has a high bandwidth.
Note:
If no port number is specified "HOST_NODE_ADDR" defaults to 29400.
Note:
The "--standalone" option can be passed to launch a single node job
  with a sidecar rendezvous backend. You donât have to pass "--
  rdzv_id", "--rdzv_endpoint", and "--rdzv_backend" when the "--
  standalone" option is used.
Note:
Learn more about writing your distributed training script here.
If "torchrun" does not meet your requirements you may use our APIs
directly for more powerful customization. Start by taking a look at