class_docstr: Assign LR based on a cyclical schedule that follows the cosine function.
  
  See https://arxiv.org/pdf/1608.03983.pdf for details.
  
  We also support a warmup phase where we linearly increase the learning rate
  from some initial learning rate (``--warmup-init-lr``) until the configured
  max learning rate (``--lr``).
  
  During warmup::
  
    lrs = torch.linspace(cfg.warmup_init_lr, cfg.lr, cfg.warmup_updates)
    lr = lrs[update_num]
  
  After warmup::
  
    lr = cfg.min_lr + 0.5*(cfg.lr - cfg.min_lr)*(1 + cos(t_curr / t_i))
  
  where ``t_curr`` is current percentage of updates within the current period
  range and ``t_i`` is the current period range, which is scaled by ``t_mul``
  after every iteration.
code: def __init__(self, cfg: CosineLRScheduleConfig, fairseq_optimizer):
      super().__init__(cfg, fairseq_optimizer)
      if isinstance(cfg.lr, Collection) and len(cfg.lr) > 1:
          raise ValueError(
              "Cannot use a fixed learning rate schedule with cosine."
              f" Consider --lr-scheduler=fixed instead. ({cfg.lr})"
          )
  
      self.max_lr = (
          cfg.lr[0] if isinstance(cfg.lr, Collection) else cfg.lr
      )
      if self.max_lr < cfg.min_lr:
          cfg.min_lr = self.max_lr
  
      warmup_end_lr = self.max_lr
      if cfg.warmup_init_lr < 0:
          cfg.warmup_init_lr = cfg.min_lr
  
      self.t_mult = cfg.t_mult
      self.period = cfg.lr_period_updates
  
      if self.period <= 0:
          assert (
              cfg.max_update > 0
          ), "Either --max_update or --lr-period-updates must be set"
          self.period = cfg.max_update - cfg.warmup_updates
  
      if cfg.warmup_updates > 0:
          # linearly warmup for the first cfg.warmup_updates
          self.lr_step = (
              warmup_end_lr - cfg.warmup_init_lr
          ) / cfg.warmup_updates
      else:
          self.lr_step = 1
  
      self.warmup_updates = cfg.warmup_updates
      self.lr_shrink = cfg.lr_shrink
  
      # initial learning rate
      self.lr = cfg.warmup_init_lr
      self.optimizer.set_lr(self.lr)
  
file_path: fairseq/fairseq/optim/lr_scheduler/cosine_lr_scheduler.py
imports: import math
  from collections.abc import Collection
  from dataclasses import dataclass, field
  from typing import List
  from omegaconf import II
  from fairseq.dataclass import FairseqDataclass
  from fairseq.optim.lr_scheduler import FairseqLRScheduler, register_lr_scheduler
label_desc: Use image node_id 1 to create a new CosineLRSchedule object from inherited base classes: FairseqLRScheduler with example: obj = CosineLRSchedule(cfg, fairseq_optimizer)