class_docstr: A high-level interface to DecentralizedAverager that averages trainable params or gradients for an optimizer.
  
  This averager implements a number of typical use cases that arise in collaborative optimization
  - averaging parameters or gradients or both (in future, this will support averaging learning rates as well)
  - this peer's weight (e.g. based on its batch size) can be specified via averager.step(weight=...)
  - when out of sync, the averager will load the entire optimizer state from an up-to-date peer
  
  :param opt: a pytorch optimizer to be averaged between peers (complete with model parameters)
  :param average_parameters: whether or not to average model parameters in self.step(...)
  :param average_gradients: whether or not to average model gradients in self.step(...)
  :param average_opt_statistics: if specified, average optimizer statistics with corresponding names in statedict
  :param initialize_optimizer: if True, this will run a speculative optimizer step with
    zero gradients to initialize all tensors. If False, please initialize the optimizer state manually.
  :param extra_tensors: if specified, these extra tensors will also be averaged and shared in load_state_from_peers.
  :note: you can use extra_tensors for averaging tensors that are updated outside of opt.step (e.g. batchnorm stats)
  :param kwargs: any additional parameters will be forwarded to DecentralizedAverager
class_docstr_tok: ["A","high-level","interface","to","DecentralizedAverager","that","averages","trainable","params","or","gradients","for","an","optimizer",".","This","averager","implements","a","number","of","typical","use","cases","that","arise","in","collaborative","optimization","-","averaging","parameters","or","gradients","or","both","(","in","future",",","this","will","support","averaging","learning","rates","as","well",")","-","this","peer","'s","weight","(","e.g",".","based","on","its","batch","size",")","can","be","specified","via","averager.step","(","weight=","...",")","-","when","out","of","sync",",","the","averager","will","load","the","entire","optimizer","state","from","an","up-to-date","peer",":","param","opt",":","a","pytorch","optimizer","to","be","averaged","between","peers","(","complete","with","model","parameters",")",":","param","average_parameters",":","whether","or","not","to","average","model","parameters","in","self.step","(","...",")",":","param","average_gradients",":","whether","or","not","to","average","model","gradients","in","self.step","(","...",")",":","param","average_opt_statistics",":","if","specified",",","average","optimizer","statistics","with","corresponding","names","in","statedict",":","param","initialize_optimizer",":","if","True",",","this","will","run","a","speculative","optimizer","step","with","zero","gradients","to","initialize","all","tensors",".","If","False",",","please","initialize","the","optimizer","state","manually",".",":","param","extra_tensors",":","if","specified",",","these","extra","tensors","will","also","be","averaged","and","shared","in","load_state_from_peers",".",":","note",":","you","can","use","extra_tensors","for","averaging","tensors","that","are","updated","outside","of","opt.step","(","e.g",".","batchnorm","stats",")",":","param","kwargs",":","any","additional","parameters","will","be","forwarded","to","DecentralizedAverager"]
docstr: Get current model/optimizer state and when requested by a newbie peer. executed in the host process.
  :returns: a tuple of (serializable_small_metadata, sequence of torch tensors)
docstr_tok: ["Get","current","model\/optimizer","state","and","when","requested","by","a","newbie","peer",".","executed","in","the","host","process",".",":","returns",":","a","tuple","of","(","serializable_small_metadata",",","sequence","of","torch","tensors",")"]
code: def get_current_state(self):
      """
      Get current model/optimizer state and when requested by a newbie peer. executed in the host process.
      :returns: a tuple of (serializable_small_metadata, sequence of torch tensors)
      """
      with torch.no_grad():
          optimized_parameters = tuple(
              param.detach().cpu()
              for param_group in self.opt.param_groups
              for param in param_group["params"]
          )
          parameter_infos = [
              CompressionInfo.from_tensor(
                  param, key=key, role=TensorRole.PARAMETER
              )
              for param, key in zip(
                  optimized_parameters, self.parameter_names
              )
          ]
          extra_tensors = tuple(
              tensor.detach().cpu() for tensor in self.extra_tensors
          )
          extra_infos = [
              CompressionInfo.from_tensor(
                  extra_tensor, key=i, role=TensorRole.UNSPECIFIED
              )
              for i, extra_tensor in enumerate(extra_tensors)
          ]
          optimizer_metadata, optimizer_tensors = dump_optimizer_state(
              self.opt
          )
          optimizer_infos = [
              CompressionInfo.from_tensor(
                  opt_tensor, key=i, role=TensorRole.OPTIMIZER
              )
              for i, opt_tensor in enumerate(optimizer_tensors)
          ]
  
      metadata = dict(
          step=self.local_step,
          group_bits=self.get_group_bits(),
          optimizer_metadata=optimizer_metadata,
      )
      all_tensors = list(
          chain(optimized_parameters, extra_tensors, optimizer_tensors)
      )
      all_tensor_infos = list(
          chain(parameter_infos, extra_infos, optimizer_infos)
      )
      return metadata, all_tensors, all_tensor_infos
  
imports: from concurrent.futures import ThreadPoolExecutor
  from contextlib import nullcontext
  from itertools import chain
  from threading import Event, Lock
  from typing import Dict, Iterator, Optional, Sequence
  import torch
  from hivemind.averaging import DecentralizedAverager
  from hivemind.compression import CompressionInfo, TensorRole
  from hivemind.utils import get_logger, nested_flatten, nested_pack
label_desc: Use image node_id 5 for calling the TrainingAverager obj's underlying member method code with example usage: obj.get_current_state() and returns: metadata,  all_tensors,  all_tensor_infos