class_docstr: Gradient Communication Quantization (GCQ) in multi-machine distributed training.
  
  It quantizes gradients to int8 locally before gradients communicate between machines.
  When the gradient communication is done, it dequantizes the int8 gradients to original dtype (default: fp16) locally.
  
  Note: When training on multiple machines, it can accelerate significantly, as multi-machine communication is the bottleneck.
  However, it is even slower when training on single machine.
  Please choose if GCQ should be on according to your training environment.
class_docstr_tok: ["Gradient","Communication","Quantization","(","GCQ",")","in","multi-machine","distributed","training",".","It","quantizes","gradients","to","int8","locally","before","gradients","communicate","between","machines",".","When","the","gradient","communication","is","done",",","it","dequantizes","the","int8","gradients","to","original","dtype","(","default",":","fp16",")","locally",".","Note",":","When","training","on","multiple","machines",",","it","can","accelerate","significantly",",","as","multi-machine","communication","is","the","bottleneck",".","However",",","it","is","even","slower","when","training","on","single","machine",".","Please","choose","if","GCQ","should","be","on","according","to","your","training","environment","."]
docstr: Calculate the scale for quantization.
docstr_tok: ["Calculate","the","scale","for","quantization","."]
code: def get_scale(self):
      """
      Calculate the scale for quantization.
      """
      # Pad original gradient if needed.
      if self.bucket_size % self.hidden_size != 0:
          rest_size = self.hidden_size - (
              self.bucket_size % self.hidden_size
          )
          self.original_gradient = torch.nn.functional.pad(
              self.original_gradient, (0, rest_size)
          )
      # Reshape original gradient to [-1, self.hidden_size].
      self.original_gradient = self.original_gradient.view(
          -1, self.hidden_size
      )
      # Calculate the max absolute value of every row in original gradient.
      max_abs_bucket = (
          self.original_gradient.abs()
          .topk(self.topk_value, dim=-1, sorted=False)[0]
          .min(dim=-1, keepdim=True)[0]
      )
      self.scale = max_abs_bucket.div(self.max_bound)
      return self.scale
  
file_path: lightseq/lightseq/training/gcq/gcq.py
imports: import math
  import torch
  import torch.distributed
label_desc: Use image node_id 2 for calling the GCQ obj's underlying member method code with example usage: obj.get_scale() and returns: self