class_docstr: Contiguous memory buffer.
  Allocate a contiguous memory of type `dtype` and size `numel`. It is
  used to reduce memory fragmentation.
  
  Usage: After the allocation, the `_start` index is set tot the first
         index of the memory. A memory chunk starting from `_start` index
         can be `allocated` for an input tensor, with the elements of the
         tensor being coppied. The buffer can be reused by resetting the
         `_start` index.
class_docstr_tok: ["Contiguous","memory","buffer",".","Allocate","a","contiguous","memory","of","type","`","dtype","`","and","size","`","numel","`",".","It","is","used","to","reduce","memory","fragmentation",".","Usage",":","After","the","allocation",",","the","`","_start","`","index","is","set","tot","the","first","index","of","the","memory",".","A","memory","chunk","starting","from","`","_start","`","index","can","be","`","allocated","`","for","an","input","tensor",",","with","the","elements","of","the","tensor","being","coppied",".","The","buffer","can","be","reused","by","resetting","the","`","_start","`","index","."]
code: def __init__(self, name, numel, dtype, track_usage):
      if torch.distributed.get_rank() == 0:
          element_size = torch.tensor([], dtype=dtype).element_size()
          print(
              "> building the {} memory buffer with {} num elements "
              "and {} dtype ({:.1f} MB)...".format(
                  name, numel, dtype, numel * element_size / 1024 / 1024
              ),
              flush=True,
          )
      self.name = name
      self.numel = numel
      self.dtype = dtype
      self.data = torch.empty(
          self.numel,
          dtype=self.dtype,
          device=torch.cuda.current_device(),
          requires_grad=False,
      )
  
      # Index tracking the start of the free memory.
      self._start = 0
  
      # Values used for tracking usage.
      self.track_usage = track_usage
      if self.track_usage:
          self.in_use_value = 0.0
          self.total_value = 0.0
  
file_path: megatron-lm/megatron/memory.py
imports: import torch
label_desc: Use image node_id 1 to create a new MemoryBuffer object  with example: obj = MemoryBuffer(name, numel, dtype, track_usage)