class_docstr: :class:`FP16_Optimizer` A cutdown version of apex.fp16_utils.FP16_Optimizer.
  Designed only to wrap apex.contrib.optimizers.FusedAdam, FusedSGD.
  Refer to apex.fp16_utils documents for more information.
  Example::
      model = torch.nn.Linear(D_in, D_out).cuda().half()
      optimizer = apex.contrib.optimizers.FusedSGD(model.parameters())
      optimizer = FP16_Optimizer(optimizer, static_loss_scale = 128.0)
      ...
      # loss.backward() becomes:
      optimizer.backward(loss)
      ...
  Example with dynamic loss scaling::
      ...
      optimizer = FP16_Optimizer(optimizer, dynamic_loss_scale=True)
                                 # optional arg to control dynamic loss scaling behavior
                                 # dynamic_loss_args={'scale_window' : 500})
                                 # Usually, dynamic_loss_args is not necessary.
class_docstr_tok: [":","class",":","`","FP16_Optimizer","`","A","cutdown","version","of","apex.fp16_utils.FP16_Optimizer",".","Designed","only","to","wrap","apex.contrib.optimizers.FusedAdam",",","FusedSGD",".","Refer","to","apex.fp16_utils","documents","for","more","information",".","Example",":",":","model","=","torch.nn.Linear","(","D_in",",","D_out",")",".cuda","(",")",".half","(",")","optimizer","=","apex.contrib.optimizers.FusedSGD","(","model.parameters","(",")",")","optimizer","=","FP16_Optimizer","(","optimizer",",","static_loss_scale","=","128.0",")","...","#","loss.backward","(",")","becomes",":","optimizer.backward","(","loss",")","...","Example","with","dynamic","loss","scaling",":",":","...","optimizer","=","FP16_Optimizer","(","optimizer",",","dynamic_loss_scale=True",")","#","optional","arg","to","control","dynamic","loss","scaling","behavior","#","dynamic_loss_args=","{","'scale_window","'",":","500","}",")","#","Usually",",","dynamic_loss_args","is","not","necessary","."]
docstr: :attr:`backward` performs the following steps:
  1. fp32_loss = loss.float()
  2. scaled_loss = fp32_loss*loss_scale
  3. scaled_loss.backward(), which accumulates scaled gradients into the ``.grad`` attributes of the model's fp16 leaves
docstr_tok: [":","attr",":","`","backward","`","performs","the","following","steps",":","1.","fp32_loss","=","loss.float","(",")","2.","scaled_loss","=","fp32_loss","*","loss_scale","3.","scaled_loss.backward","(",")",",","which","accumulates","scaled","gradients","into","the","``",".grad","``","attributes","of","the","model","'s","fp16","leaves"]
code: def backward(self, loss):
      """
      :attr:`backward` performs the following steps:
      1. fp32_loss = loss.float()
      2. scaled_loss = fp32_loss*loss_scale
      3. scaled_loss.backward(), which accumulates scaled gradients into the ``.grad`` attributes of the model's fp16 leaves
      """
      scaled_loss = (loss.float()) * self.cur_scale
      scaled_loss.backward()
  
file_path: apex/apex/contrib/optimizers/fp16_optimizer.py
imports: import torch
  from apex.multi_tensor_apply import multi_tensor_applier
label_desc: Use image node_id 4 for calling the FP16_Optimizer obj's underlying member method code with example usage: obj.backward(loss) without return types