title: In distributed computing, what are world size and rank?
body: <p>I've been reading through <a href="https://yangkky.github.io/2019/07/08/distributed-pytorch-tutorial.html" rel="noreferrer">some documentation</a> and <a href="https://github.com/narumiruna/pytorch-distributed-example/blob/master/toy/main.py" rel="noreferrer">example code</a> with the end goal of writing scripts for distributed computing (running PyTorch), but the concepts confuse me.</p>
  
  <p>Let's assume that we have a single node with 4 GPUs, and we want to run our script on those 4 GPUs (i.e. one process per GPU). In such a scenario, what are the rank world size and rank? I often find the explanation for world size: <em>Total number of processes involved in the job</em>, so I assume that that is four in our example, but what about rank?</p>
  
  <p>To explain it further, another example with multiple nodes and multiple GPUs could be useful, too.</p>