Train script
If your train script works with "torch.distributed.launch" it will
continue working with "torchrun" with these differences:


No need to manually pass "RANK", "WORLD_SIZE", "MASTER_ADDR", and
   "MASTER_PORT".


"rdzv_backend" and "rdzv_endpoint" can be provided. For most users
   this will be set to "c10d" (see rendezvous). The default
   "rdzv_backend" creates a non-elastic rendezvous where
   "rdzv_endpoint" holds the master address.


Make sure you have a "load_checkpoint(path)" and
   "save_checkpoint(path)" logic in your script. When any number of
   workers fail we restart all the workers with the same program
   arguments so you will lose progress up to the most recent
   checkpoint (see elastic launch).


"use_env" flag has been removed. If you were parsing local rank by
   parsing the "--local_rank" option, you need to get the local rank
   from the environment variable "LOCAL_RANK" (e.g.
   "int(os.environ["LOCAL_RANK"])").

