class_docstr: CrossEntropy loss is calculated in parallel, allreduce denominator into single gpu and calculate softmax.
  Implemented of ArcFace (https://arxiv.org/pdf/1801.07698v1.pdf):
class_docstr_tok: ["CrossEntropy","loss","is","calculated","in","parallel",",","allreduce","denominator","into","single","gpu","and","calculate","softmax",".","Implemented","of","ArcFace","(","https",":","\/\/arxiv.org\/pdf\/1801.07698v1.pdf",")",":"]
docstr: Args:
      loss_grad (torch.Tensor): gradient backward by last layer
  Returns:
      gradients for each input in forward function
      `None` gradients for one-hot label
docstr_tok: ["Args",":","loss_grad","(","torch.Tensor",")",":","gradient","backward","by","last","layer","Returns",":","gradients","for","each","input","in","forward","function","`","None","`","gradients","for","one-hot","label"]
code: def backward(ctx, loss_gradient):
      """
      Args:
          loss_grad (torch.Tensor): gradient backward by last layer
      Returns:
          gradients for each input in forward function
          `None` gradients for one-hot label
      """
      (
          index,
          logits,
          label,
      ) = ctx.saved_tensors
      batch_size = logits.size(0)
      one_hot = torch.zeros(
          size=[index.size(0), logits.size(1)], device=logits.device
      )
      one_hot.scatter_(1, label[index], 1)
      logits[index] -= one_hot
      logits.div_(batch_size)
      return logits * loss_gradient.item(), None
  
imports: import math
  from typing import Callable
  import torch
  from torch import distributed
  from torch.nn.functional import linear, normalize
label_desc: Use image node_id 2 for calling the DistCrossEntropyFunc obj's underlying member method code with example usage: obj.backward(ctx, loss_gradient) and returns: unknown,  None