class_docstr: Decay the LR based on the inverse square root of the update number.
  
  We also support a warmup phase where we linearly increase the learning rate
  from some initial learning rate (``--warmup-init-lr``) until the configured
  learning rate (``--lr``). Thereafter we decay proportional to the number of
  updates, with a decay factor set to align with the configured learning rate.
  
  During warmup::
  
    lrs = torch.linspace(cfg.warmup_init_lr, cfg.lr, cfg.warmup_updates)
    lr = lrs[update_num]
  
  After warmup::
  
    decay_factor = cfg.lr * sqrt(cfg.warmup_updates)
    lr = decay_factor / sqrt(update_num)
class_docstr_tok: ["Decay","the","LR","based","on","the","inverse","square","root","of","the","update","number",".","We","also","support","a","warmup","phase","where","we","linearly","increase","the","learning","rate","from","some","initial","learning","rate","(","``","--","warmup-init-lr","``",")","until","the","configured","learning","rate","(","``","--","lr","``",")",".","Thereafter","we","decay","proportional","to","the","number","of","updates",",","with","a","decay","factor","set","to","align","with","the","configured","learning","rate",".","During","warmup",":",":","lrs","=","torch.linspace","(","cfg.warmup_init_lr",",","cfg.lr",",","cfg.warmup_updates",")","lr","=","lrs","[","update_num","]","After","warmup",":",":","decay_factor","=","cfg.lr","*","sqrt","(","cfg.warmup_updates",")","lr","=","decay_factor","\/","sqrt","(","update_num",")"]
docstr: Update the learning rate at the end of the given epoch.
docstr_tok: ["Update","the","learning","rate","at","the","end","of","the","given","epoch","."]
code: def step(self, epoch, val_loss=None):
      """Update the learning rate at the end of the given epoch."""
      super().step(epoch, val_loss)
      # we don't change the learning rate at epoch boundaries
      return self.optimizer.get_lr()
  
file_path: fairseq/fairseq/optim/lr_scheduler/inverse_square_root_schedule.py
imports: from collections.abc import Collection
  from dataclasses import dataclass, field
  from typing import List
  from omegaconf import II
  from fairseq.dataclass import FairseqDataclass
  from fairseq.optim.lr_scheduler import FairseqLRScheduler, register_lr_scheduler
label_desc: Use image node_id 2 for calling the InverseSquareRootSchedule obj's underlying member method code with example usage: obj.step(epoch, val_loss) and returns: self