this context manager is not aware of non-DDP collective
        communication. This flag will cause all ranks to throw when
        any one rank exhausts inputs, allowing these errors to be
        caught and recovered from across all ranks.
  Parameters:
     * **divide_by_initial_world_size** (*bool*) -- If "True",
       will divide gradients by the initial "world_size" DDP
       training was launched with. If "False", will compute the
       effective world size (number of ranks that have not
       depleted their inputs yet) and divide gradients by that
       during allreduce. Set "divide_by_initial_world_size=True"
       to ensure every input sample including the uneven inputs
       have equal weight in terms of how much they contribute to
       the global gradient. This is achieved by always dividing
       the gradient by the initial "world_size" even when we
       encounter uneven inputs. If you set this to "False", we
