class_docstr: ## Sophia-G Optimizer
  
  We extend the class `GenericAdaptiveOptimizer` defined in [`__init__.py`](index.html)
  to implement the Sophia optimizer.
docstr: ### Initialize the optimizer
  
  * `params` is the list of parameters
  * `lr` is the maximum learning rate $\eta 
  ho$
  * `betas` is a tuple of ($eta_1$, $eta_2$)
  * `eps` is $\epsilon$
  * `pho` is $
  ho$
  * `weight_decay` is an instance of class `WeightDecay` defined in [`__init__.py`](index.html)
  * `defaults` is a dictionary of default for group values.
   This is useful when you want to extend the class `Adam`.
code: def __init__(
      self,
      params,
      lr: float = 1e-4,
      betas: Tuple[float, float] = (0.9, 0.95),
      eps: float = 1e-12,
      rho: float = 0.03,
      weight_decay: WeightDecay = WeightDecay(),
      defaults: Optional[Dict[str, Any]] = None,
  ):
      """
      ### Initialize the optimizer
  
      * `params` is the list of parameters
      * `lr` is the maximum learning rate $\eta \rho$
      * `betas` is a tuple of ($\beta_1$, $\beta_2$)
      * `eps` is $\epsilon$
      * `pho` is $\rho$
      * `weight_decay` is an instance of class `WeightDecay` defined in [`__init__.py`](index.html)
      * `defaults` is a dictionary of default for group values.
       This is useful when you want to extend the class `Adam`.
      """
      defaults = {} if defaults is None else defaults
      defaults.update(weight_decay.defaults())
      defaults.update(dict(rho=rho))
      super().__init__(params, defaults, lr, betas, eps)
  
      self.weight_decay = weight_decay
  
file_path: annotated_deep_learning_paper_implementations/labml_nn/optimizers/sophia.py
imports: from typing import Dict, Any, Tuple, Optional
  import torch
  from torch import nn
  from labml_nn.optimizers import GenericAdaptiveOptimizer, WeightDecay
label_desc: Use image node_id 1 to create a new Sophia object from inherited base classes: GenericAdaptiveOptimizer with example: obj = Sophia(params, lr, betas, eps, rho, weight_decay, defaults)