class_docstr: Retrospective Adversarial Replay for Continual Learning
  https://openreview.net/forum?id=XEoih0EwCwL
  Continual learning is an emerging research challenge in machine learning
  that addresses the problem where models quickly fit the most recently
  trained-on data and are prone to catastrophic forgetting due to
  distribution shifts --- it does this by maintaining a small historical
  replay buffer in replay-based methods.
  To avoid these problems, this paper proposes a method,
  ``Retrospective Adversarial Replay (RAR)'', that synthesizes adversarial
  samples near the forgetting boundary. RAR perturbs a buffered sample
  towards its nearest neighbor drawn from the current task in a latent
  representation space. By replaying such samples, we are able to refine the
  boundary between previous and current tasks, hence combating forgetting and
  reducing bias towards the current task. To mitigate the severity of a small
  replay buffer, we develop a novel MixUp-based strategy to increase replay
  variation by replaying mixed augmentations.
  Combined with RAR, this achieves a holistic framework that helps to
  alleviate catastrophic forgetting. We show that this excels on
  broadly-used benchmarks and outperforms other continual learning baselines
  especially when only a small buffer is used. We conduct a thorough
  ablation study over each key component as well as a hyperparameter
  sensitivity analysis to demonstrate the effectiveness and robustness of RAR.
class_docstr_tok: ["Retrospective","Adversarial","Replay","for","Continual","Learning","https",":","\/\/openreview.net\/forum","?","id=XEoih0EwCwL","Continual","learning","is","an","emerging","research","challenge","in","machine","learning","that","addresses","the","problem","where","models","quickly","fit","the","most","recently","trained-on","data","and","are","prone","to","catastrophic","forgetting","due","to","distribution","shifts","--","-","it","does","this","by","maintaining","a","small","historical","replay","buffer","in","replay-based","methods",".","To","avoid","these","problems",",","this","paper","proposes","a","method",",","``","Retrospective","Adversarial","Replay","(","RAR",")","''",",","that","synthesizes","adversarial","samples","near","the","forgetting","boundary",".","RAR","perturbs","a","buffered","sample","towards","its","nearest","neighbor","drawn","from","the","current","task","in","a","latent","representation","space",".","By","replaying","such","samples",",","we","are","able","to","refine","the","boundary","between","previous","and","current","tasks",",","hence","combating","forgetting","and","reducing","bias","towards","the","current","task",".","To","mitigate","the","severity","of","a","small","replay","buffer",",","we","develop","a","novel","MixUp-based","strategy","to","increase","replay","variation","by","replaying","mixed","augmentations",".","Combined","with","RAR",",","this","achieves","a","holistic","framework","that","helps","to","alleviate","catastrophic","forgetting",".","We","show","that","this","excels","on","broadly-used","benchmarks","and","outperforms","other","continual","learning","baselines","especially","when","only","a","small","buffer","is","used",".","We","conduct","a","thorough","ablation","study","over","each","key","component","as","well","as","a","hyperparameter","sensitivity","analysis","to","demonstrate","the","effectiveness","and","robustness","of","RAR","."]
docstr: Before the backward function in the training process. We need to update
  the loss function.
  
  First we add the loss of the buffer. Then if we are using RAR, we
  obtained the features to find the closest element of a different class.
  We apply FGSM to those selected elements and then find the loss
  of the attacked samples.
docstr_tok: ["Before","the","backward","function","in","the","training","process",".","We","need","to","update","the","loss","function",".","First","we","add","the","loss","of","the","buffer",".","Then","if","we","are","using","RAR",",","we","obtained","the","features","to","find","the","closest","element","of","a","different","class",".","We","apply","FGSM","to","those","selected","elements","and","then","find","the","loss","of","the","attacked","samples","."]
code: def before_backward(self, strategy: "SupervisedTemplate", **kwargs):
      """
      Before the backward function in the training process. We need to update
      the loss function.
  
      First we add the loss of the buffer. Then if we are using RAR, we
      obtained the features to find the closest element of a different class.
      We apply FGSM to those selected elements and then find the loss
      of the attacked samples.
      """
  
      if len(self.storage_policy.buffer) == 0:
          # first experience. We don't use the buffer
          return
  
      batch_buff = self.get_buffer_batch()
      mb_x_buff = batch_buff[0].to(strategy.device)
      mb_y_buff = batch_buff[1].to(strategy.device)
  
      # out_buff = strategy.model(mb_x_buff)
      # strategy.loss += (1-self.beta_coef) * \
      #     strategy._criterion(out_buff, mb_y_buff)
  
      if not self.use_adversarial_replay:
          return
  
      if self.name_ext_layer is None:
          self.name_ext_layer = get_graph_node_names(strategy.model)[0][
              -2
          ]
  
      copy_model = copy.deepcopy(strategy.model)
      feature_extractor = create_feature_extractor(
          copy_model, return_nodes=[self.name_ext_layer]
      )
  
      optimizer = SGD(copy_model.parameters(), lr=self.opt_lr)
  
      optimizer.zero_grad()
      output = copy_model(strategy.mb_x)
      loss = strategy._criterion(output, strategy.mb_y)
      loss.backward()
      optimizer.step()
  
      out_curr = feature_extractor(strategy.mb_x)[self.name_ext_layer]
      out_buff = feature_extractor(mb_x_buff)[self.name_ext_layer]
  
      dist = torch.cdist(out_buff, out_curr)
      _, ind = torch.sort(dist)
  
      target_attack = (
          torch.zeros(dist.size(0)).long().to(strategy.device)
      )
      for j in range(dist.size(0)):
          for i in ind[j]:
              if mb_y_buff[j].item() != strategy.mb_y[i].item():
                  target_attack[j] = strategy.mb_y[i]
  
      mb_x_buff.requires_grad = True
      out_pert = copy_model(mb_x_buff)
      loss = strategy._criterion(out_pert, target_attack)
      loss.backward()
  
      mb_x_pert = self.mifgsm_attack(mb_x_buff, mb_x_buff.grad.data)
  
      out_buff = strategy.model(mb_x_pert)
      strategy.loss += (self.beta_coef) * strategy._criterion(
          out_buff, mb_y_buff
      )
  
file_path: avalanche/avalanche/training/plugins/rar.py
imports: import copy
  from typing import TYPE_CHECKING, Optional
  import numpy
  import torch
  from torch.optim import SGD
  from torch.utils.data.dataloader import DataLoader
  from torchvision.models.feature_extraction import get_graph_node_names, create_feature_extractor
  from avalanche.training.plugins.strategy_plugin import SupervisedPlugin
  from avalanche.training.storage_policy import ReservoirSamplingBuffer
  from avalanche.benchmarks.utils.data_loader import ReplayDataLoader
label_desc: Use image node_id 3 for calling the RARPlugin obj's underlying member method code with example usage: obj.before_backward(strategy) without return types