class_docstr: A high-level interface to DecentralizedAverager that averages trainable params or gradients for an optimizer.
  
  This averager implements a number of typical use cases that arise in collaborative optimization
  - averaging parameters or gradients or both (in future, this will support averaging learning rates as well)
  - this peer's weight (e.g. based on its batch size) can be specified via averager.step(weight=...)
  - when out of sync, the averager will load the entire optimizer state from an up-to-date peer
  
  :param opt: a pytorch optimizer to be averaged between peers (complete with model parameters)
  :param average_parameters: whether or not to average model parameters in self.step(...)
  :param average_gradients: whether or not to average model gradients in self.step(...)
  :param average_opt_statistics: if specified, average optimizer statistics with corresponding names in statedict
  :param initialize_optimizer: if True, this will run a speculative optimizer step with
    zero gradients to initialize all tensors. If False, please initialize the optimizer state manually.
  :param extra_tensors: if specified, these extra tensors will also be averaged and shared in load_state_from_peers.
  :note: you can use extra_tensors for averaging tensors that are updated outside of opt.step (e.g. batchnorm stats)
  :param kwargs: any additional parameters will be forwarded to DecentralizedAverager
class_docstr_tok: ["A","high-level","interface","to","DecentralizedAverager","that","averages","trainable","params","or","gradients","for","an","optimizer",".","This","averager","implements","a","number","of","typical","use","cases","that","arise","in","collaborative","optimization","-","averaging","parameters","or","gradients","or","both","(","in","future",",","this","will","support","averaging","learning","rates","as","well",")","-","this","peer","'s","weight","(","e.g",".","based","on","its","batch","size",")","can","be","specified","via","averager.step","(","weight=","...",")","-","when","out","of","sync",",","the","averager","will","load","the","entire","optimizer","state","from","an","up-to-date","peer",":","param","opt",":","a","pytorch","optimizer","to","be","averaged","between","peers","(","complete","with","model","parameters",")",":","param","average_parameters",":","whether","or","not","to","average","model","parameters","in","self.step","(","...",")",":","param","average_gradients",":","whether","or","not","to","average","model","gradients","in","self.step","(","...",")",":","param","average_opt_statistics",":","if","specified",",","average","optimizer","statistics","with","corresponding","names","in","statedict",":","param","initialize_optimizer",":","if","True",",","this","will","run","a","speculative","optimizer","step","with","zero","gradients","to","initialize","all","tensors",".","If","False",",","please","initialize","the","optimizer","state","manually",".",":","param","extra_tensors",":","if","specified",",","these","extra","tensors","will","also","be","averaged","and","shared","in","load_state_from_peers",".",":","note",":","you","can","use","extra_tensors","for","averaging","tensors","that","are","updated","outside","of","opt.step","(","e.g",".","batchnorm","stats",")",":","param","kwargs",":","any","additional","parameters","will","be","forwarded","to","DecentralizedAverager"]
docstr: Attempt to download the latest optimizer state from peers and update trainer parameters/statistics.
  :returns: whether or the averager succeeded in loading parameters
docstr_tok: ["Attempt","to","download","the","latest","optimizer","state","from","peers","and","update","trainer","parameters\/statistics",".",":","returns",":","whether","or","the","averager","succeeded","in","loading","parameters"]
code: def load_state_from_peers(self, **kwargs):
      """
      Attempt to download the latest optimizer state from peers and update trainer parameters/statistics.
      :returns: whether or the averager succeeded in loading parameters
      """
      parameters_and_extras = [
          param
          for param_group in self.opt.param_groups
          for param in param_group["params"]
      ]
      parameters_and_extras.extend(self.extra_tensors)
      num_local_tensors = len(parameters_and_extras)
  
      loaded_state = super().load_state_from_peers(**kwargs)
      if loaded_state is None:
          return
      metadata, flat_tensors = loaded_state
      loaded_parameters_and_extras = flat_tensors[:num_local_tensors]
      loaded_opt_tensors = flat_tensors[num_local_tensors:]
  
      with torch.no_grad():
          for local_param, loaded_param in zip(
              parameters_and_extras, loaded_parameters_and_extras
          ):
              local_param[...] = loaded_param
          load_optimizer_state(
              self.opt,
              metadata["optimizer_metadata"],
              loaded_opt_tensors,
          )
  
      self.local_step = max(self.local_step, metadata["step"])
  
imports: from concurrent.futures import ThreadPoolExecutor
  from contextlib import nullcontext
  from itertools import chain
  from threading import Event, Lock
  from typing import Dict, Iterator, Optional, Sequence
  import torch
  from hivemind.averaging import DecentralizedAverager
  from hivemind.compression import CompressionInfo, TensorRole
  from hivemind.utils import get_logger, nested_flatten, nested_pack
label_desc: Use image node_id 6 for calling the TrainingAverager obj's underlying member method code with example usage: obj.load_state_from_peers() without return types