class_docstr: Allows measuring model robustness for a given attack or set of attacks. This class
  can be used with any metric(s) as well as any set of attacks, either based on
  attacks / perturbations from captum.robust such as FGSM or PGD or external
  augmentation methods or perturbations such as torchvision transforms.
docstr: Evaluate model and attack performance on provided inputs
  
  Args:
  
      inputs (Any): Input for which attack metrics
          are computed. It can be provided as a tensor, tuple of tensors,
          or any raw input type (e.g. PIL image or text string).
          This input is provided directly as input to preproc function as well
          as any attack applied before preprocessing. If no pre-processing
          function is provided, this input is provided directly to the main
          model and all attacks.
  
      additional_forward_args (Any, optional): If the forward function
          requires additional arguments other than the preprocessing
          outputs (or inputs if preproc_fn is None), this argument
          can be provided. It must be either a single additional
          argument of a Tensor or arbitrary (non-tuple) type or a
          tuple containing multiple additional arguments including
          tensors or any arbitrary python types. These arguments
          are provided to forward_func in order following the
          arguments in inputs.
          For a tensor, the first dimension of the tensor must
          correspond to the number of examples. For all other types,
          the given argument is used for all forward evaluations.
          Default: ``None``
      perturbations_per_eval (int, optional): Allows perturbations of multiple
          attacks to be grouped and evaluated in one call of forward_fn
          Each forward pass will contain a maximum of
          perturbations_per_eval * #examples samples.
          For DataParallel models, each batch is split among the
          available devices, so evaluations on each available
          device contain at most
          (perturbations_per_eval * #examples) / num_devices
          samples.
          In order to apply this functionality, the output of preproc_fn
          (or inputs itself if no preproc_fn is provided) must be a tensor
          or tuple of tensors.
          Default: ``1``
      kwargs (Any, optional): Additional keyword arguments provided to metric
          function as well as selected attacks based on chosen additional_args.
          Default: ``None``
  
  Returns:
  
  - **attack results** Dict: str -> Dict[str, Union[Tensor, Tuple[Tensor, ...]]]:
          Dictionary containing attack results for provided batch.
          Maps attack name to dictionary,
          containing best-case, worst-case and average-case results for attack.
          Dictionary contains keys "mean", "max" and "min" when num_attempts > 1
          and only "mean" for num_attempts = 1, which contains the (single) metric
          result for the attack attempt.
          An additional key of 'Original' is included with metric results
          without any perturbations.
  
  
  Examples::
  
  >>> def accuracy_metric(model_out: Tensor, targets: Tensor):
  >>>     return torch.argmax(model_out, dim=1) == targets).float()
  
  >>> attack_metric = AttackComparator(model=resnet18,
                                       metric=accuracy_metric,
                                       preproc_fn=normalize)
  
  >>> random_rotation = transforms.RandomRotation()
  >>> jitter = transforms.ColorJitter()
  
  >>> attack_metric.add_attack(random_rotation, "Random Rotation",
  >>>                          num_attempts = 5)
  >>> attack_metric.add_attack((jitter, "Jitter", num_attempts = 1)
  >>> attack_metric.add_attack(FGSM(resnet18), "FGSM 0.1", num_attempts = 1,
  >>>                          apply_before_preproc=False,
  >>>                          attack_kwargs={epsilon: 0.1},
  >>>                          additional_args=["targets"])
  
  >>> for images, labels in dataloader:
  >>>     batch_results = attack_metric.evaluate(inputs=images, targets=labels)
docstr_tok: ["Evaluate","model","and","attack","performance","on","provided","inputs","Args",":","inputs","(","Any",")",":","Input","for","which","attack","metrics","are","computed",".","It","can","be","provided","as","a","tensor",",","tuple","of","tensors",",","or","any","raw","input","type","(","e.g",".","PIL","image","or","text","string",")",".","This","input","is","provided","directly","as","input","to","preproc","function","as","well","as","any","attack","applied","before","preprocessing",".","If","no","pre-processing","function","is","provided",",","this","input","is","provided","directly","to","the","main","model","and","all","attacks",".","additional_forward_args","(","Any",",","optional",")",":","If","the","forward","function","requires","additional","arguments","other","than","the","preprocessing","outputs","(","or","inputs","if","preproc_fn","is","None",")",",","this","argument","can","be","provided",".","It","must","be","either","a","single","additional","argument","of","a","Tensor","or","arbitrary","(","non-tuple",")","type","or","a","tuple","containing","multiple","additional","arguments","including","tensors","or","any","arbitrary","python","types",".","These","arguments","are","provided","to","forward_func","in","order","following","the","arguments","in","inputs",".","For","a","tensor",",","the","first","dimension","of","the","tensor","must","correspond","to","the","number","of","examples",".","For","all","other","types",",","the","given","argument","is","used","for","all","forward","evaluations",".","Default",":","``","None","``","perturbations_per_eval","(","int",",","optional",")",":","Allows","perturbations","of","multiple","attacks","to","be","grouped","and","evaluated","in","one","call","of","forward_fn","Each","forward","pass","will","contain","a","maximum","of","perturbations_per_eval","*","#","examples","samples",".","For","DataParallel","models",",","each","batch","is","split","among","the","available","devices",",","so","evaluations","on","each","available","device","contain","at","most","(","perturbations_per_eval","*","#","examples",")","\/","num_devices","samples",".","In","order","to","apply","this","functionality",",","the","output","of","preproc_fn","(","or","inputs","itself","if","no","preproc_fn","is","provided",")","must","be","a","tensor","or","tuple","of","tensors",".","Default",":","``","1","``","kwargs","(","Any",",","optional",")",":","Additional","keyword","arguments","provided","to","metric","function","as","well","as","selected","attacks","based","on","chosen","additional_args",".","Default",":","``","None","``","Returns",":","-","*","*","attack","results","*","*","Dict",":","str","-",">","Dict","[","str",",","Union","[","Tensor",",","Tuple","[","Tensor",",","...","]","]","]",":","Dictionary","containing","attack","results","for","provided","batch",".","Maps","attack","name","to","dictionary",",","containing","best-case",",","worst-case","and","average-case","results","for","attack",".","Dictionary","contains","keys","``","mean","''",",","``","max","''","and","``","min","''","when","num_attempts",">","1","and","only","``","mean","''","for","num_attempts","=","1",",","which","contains","the","(","single",")","metric","result","for","the","attack","attempt",".","An","additional","key","of","'Original","'","is","included","with","metric","results","without","any","perturbations",".","Examples",":",":",">",">",">","def","accuracy_metric","(","model_out",":","Tensor",",","targets",":","Tensor",")",":",">",">",">","return","torch.argmax","(","model_out",",","dim=1",")","==","targets",")",".float","(",")",">",">",">","attack_metric","=","AttackComparator","(","model=resnet18",",","metric=accuracy_metric",",","preproc_fn=normalize",")",">",">",">","random_rotation","=","transforms.RandomRotation","(",")",">",">",">","jitter","=","transforms.ColorJitter","(",")",">",">",">","attack_metric.add_attack","(","random_rotation",",","``","Random","Rotation","''",",",">",">",">","num_attempts","=","5",")",">",">",">","attack_metric.add_attack","(","(","jitter",",","``","Jitter","''",",","num_attempts","=","1",")",">",">",">","attack_metric.add_attack","(","FGSM","(","resnet18",")",",","``","FGSM","0.1","''",",","num_attempts","=","1",",",">",">",">","apply_before_preproc=False",",",">",">",">","attack_kwargs=","{","epsilon",":","0.1","}",",",">",">",">","additional_args=","[","``","targets","''","]",")",">",">",">","for","images",",","labels","in","dataloader",":",">",">",">","batch_results","=","attack_metric.evaluate","(","inputs=images",",","targets=labels",")"]
code: def evaluate(
      self,
      inputs: Any,
      additional_forward_args: Any = None,
      perturbations_per_eval: int = 1,
      **kwargs,
  ) -> Dict[str, Union[MetricResultType, Dict[str, MetricResultType]]]:
      r"""
      Evaluate model and attack performance on provided inputs
  
      Args:
  
          inputs (Any): Input for which attack metrics
              are computed. It can be provided as a tensor, tuple of tensors,
              or any raw input type (e.g. PIL image or text string).
              This input is provided directly as input to preproc function as well
              as any attack applied before preprocessing. If no pre-processing
              function is provided, this input is provided directly to the main
              model and all attacks.
  
          additional_forward_args (Any, optional): If the forward function
              requires additional arguments other than the preprocessing
              outputs (or inputs if preproc_fn is None), this argument
              can be provided. It must be either a single additional
              argument of a Tensor or arbitrary (non-tuple) type or a
              tuple containing multiple additional arguments including
              tensors or any arbitrary python types. These arguments
              are provided to forward_func in order following the
              arguments in inputs.
              For a tensor, the first dimension of the tensor must
              correspond to the number of examples. For all other types,
              the given argument is used for all forward evaluations.
              Default: ``None``
          perturbations_per_eval (int, optional): Allows perturbations of multiple
              attacks to be grouped and evaluated in one call of forward_fn
              Each forward pass will contain a maximum of
              perturbations_per_eval * #examples samples.
              For DataParallel models, each batch is split among the
              available devices, so evaluations on each available
              device contain at most
              (perturbations_per_eval * #examples) / num_devices
              samples.
              In order to apply this functionality, the output of preproc_fn
              (or inputs itself if no preproc_fn is provided) must be a tensor
              or tuple of tensors.
              Default: ``1``
          kwargs (Any, optional): Additional keyword arguments provided to metric
              function as well as selected attacks based on chosen additional_args.
              Default: ``None``
  
      Returns:
  
      - **attack results** Dict: str -> Dict[str, Union[Tensor, Tuple[Tensor, ...]]]:
              Dictionary containing attack results for provided batch.
              Maps attack name to dictionary,
              containing best-case, worst-case and average-case results for attack.
              Dictionary contains keys "mean", "max" and "min" when num_attempts > 1
              and only "mean" for num_attempts = 1, which contains the (single) metric
              result for the attack attempt.
              An additional key of 'Original' is included with metric results
              without any perturbations.
  
  
      Examples::
  
      >>> def accuracy_metric(model_out: Tensor, targets: Tensor):
      >>>     return torch.argmax(model_out, dim=1) == targets).float()
  
      >>> attack_metric = AttackComparator(model=resnet18,
                                           metric=accuracy_metric,
                                           preproc_fn=normalize)
  
      >>> random_rotation = transforms.RandomRotation()
      >>> jitter = transforms.ColorJitter()
  
      >>> attack_metric.add_attack(random_rotation, "Random Rotation",
      >>>                          num_attempts = 5)
      >>> attack_metric.add_attack((jitter, "Jitter", num_attempts = 1)
      >>> attack_metric.add_attack(FGSM(resnet18), "FGSM 0.1", num_attempts = 1,
      >>>                          apply_before_preproc=False,
      >>>                          attack_kwargs={epsilon: 0.1},
      >>>                          additional_args=["targets"])
  
      >>> for images, labels in dataloader:
      >>>     batch_results = attack_metric.evaluate(inputs=images, targets=labels)
      """
      additional_forward_args = _format_additional_forward_args(
          additional_forward_args
      )
      expanded_additional_args = (
          _expand_additional_forward_args(
              additional_forward_args, perturbations_per_eval
          )
          if perturbations_per_eval > 1
          else additional_forward_args
      )
  
      preproc_input = None
      if self.preproc_fn is not None:
          preproc_input = self.preproc_fn(inputs)
      else:
          preproc_input = inputs
  
      input_list = [preproc_input]
      key_list = [ORIGINAL_KEY]
  
      batch_summarizers = {ORIGINAL_KEY: Summarizer([Mean()])}
      if ORIGINAL_KEY not in self.summary_results:
          self.summary_results[ORIGINAL_KEY] = Summarizer(
              [stat() for stat in self.aggregate_stats]
          )
  
      def _check_and_evaluate(input_list, key_list):
          if len(input_list) == perturbations_per_eval:
              self._evaluate_batch(
                  input_list,
                  expanded_additional_args,
                  key_list,
                  batch_summarizers,
                  kwargs,
              )
              return [], []
          return input_list, key_list
  
      input_list, key_list = _check_and_evaluate(input_list, key_list)
  
      for attack_key in self.attacks:
          attack = self.attacks[attack_key]
          if attack.num_attempts > 1:
              stats = [stat() for stat in self.batch_stats]
          else:
              stats = [Mean()]
          batch_summarizers[attack.name] = Summarizer(stats)
          additional_attack_args = {}
          for key in attack.additional_args:
              if key not in kwargs:
                  warnings.warn(
                      f"Additional sample arg {key} not provided for {attack_key}"
                  )
              else:
                  additional_attack_args[key] = kwargs[key]
  
          for _ in range(attack.num_attempts):
              if attack.apply_before_preproc:
                  attacked_inp = attack.attack_fn(
                      inputs,
                      **additional_attack_args,
                      **attack.attack_kwargs,
                  )
                  preproc_attacked_inp = (
                      self.preproc_fn(attacked_inp)
                      if self.preproc_fn
                      else attacked_inp
                  )
              else:
                  preproc_attacked_inp = attack.attack_fn(
                      preproc_input,
                      **additional_attack_args,
                      **attack.attack_kwargs,
                  )
  
              input_list.append(preproc_attacked_inp)
              key_list.append(attack.name)
  
              input_list, key_list = _check_and_evaluate(
                  input_list, key_list
              )
  
      if len(input_list) > 0:
          final_add_args = _expand_additional_forward_args(
              additional_forward_args, len(input_list)
          )
          self._evaluate_batch(
              input_list,
              final_add_args,
              key_list,
              batch_summarizers,
              kwargs,
          )
  
      return self._parse_and_update_results(batch_summarizers)
  
imports: import warnings
  from collections import namedtuple
  from typing import Any, Callable, cast, Dict, Generic, List, NamedTuple, Optional, Tuple, TypeVar, Union
  from captum._utils.common import _expand_additional_forward_args, _format_additional_forward_args, _reduce_list
  from captum.attr import Max, Mean, Min, Summarizer
  from captum.log import log_usage
  from captum.robust._core.perturbation import Perturbation
  from torch import Tensor
label_desc: Use image node_id 6 for calling the AttackComparator obj's underlying member method code with example usage: obj.evaluate(inputs, additional_forward_args, perturbations_per_eval) and returns: self, input_list,  key_list, list,  list