class_docstr: Gradient Communication Quantization (GCQ) in multi-machine distributed training.
  
  It quantizes gradients to int8 locally before gradients communicate between machines.
  When the gradient communication is done, it dequantizes the int8 gradients to original dtype (default: fp16) locally.
  
  Note: When training on multiple machines, it can accelerate significantly, as multi-machine communication is the bottleneck.
  However, it is even slower when training on single machine.
  Please choose if GCQ should be on according to your training environment.
class_docstr_tok: ["Gradient","Communication","Quantization","(","GCQ",")","in","multi-machine","distributed","training",".","It","quantizes","gradients","to","int8","locally","before","gradients","communicate","between","machines",".","When","the","gradient","communication","is","done",",","it","dequantizes","the","int8","gradients","to","original","dtype","(","default",":","fp16",")","locally",".","Note",":","When","training","on","multiple","machines",",","it","can","accelerate","significantly",",","as","multi-machine","communication","is","the","bottleneck",".","However",",","it","is","even","slower","when","training","on","single","machine",".","Please","choose","if","GCQ","should","be","on","according","to","your","training","environment","."]
code: def __init__(
      self,
      world_size,
      hidden_size=1024,
      bucket=None,
      quantize_bits=8,
      quantile_value=0.99,
  ):
      self.world_size = world_size
      self.bucket = bucket
      self.quantize_bits = quantize_bits
      self.device = bucket.buffer().device
      self.hidden_size = hidden_size
      self.quantile_value = quantile_value
      # Actually, we use topk implementation to estimate quantile,
      # as the former is faster than the latter.
      self.topk_value = math.ceil(
          self.hidden_size * (1.0 - self.quantile_value)
      )
  
      # We use symmetrical range, i.e., [-127, 127] instead of [-128, 127], when quantize_bits is 8.
      self.max_bound = (
          torch.pow(
              torch.tensor(
                  2.0, dtype=torch.float16, device=self.device
              ),
              self.quantize_bits - 1,
          )
          - 1.0
      )
      self.min_bound = -self.max_bound
      # The new range after dividing the world_size.
      self.new_max_bound = self.max_bound.div(
          self.world_size, rounding_mode="trunc"
      )
      self.new_min_bound = self.min_bound.div(
          self.world_size, rounding_mode="trunc"
      )
  
      self.original_gradient = self.bucket.buffer()
      self.bucket_size = self.original_gradient.numel()
      self.scale = None
      self.encoded_gradient = None
  
file_path: lightseq/lightseq/training/gcq/gcq.py
imports: import math
  import torch
  import torch.distributed
label_desc: Use image node_id 1 to create a new GCQ object from inherited base classes: object with example: obj = GCQ(world_size, hidden_size, bucket, quantize_bits, quantile_value)