class_docstr: An abstract evaluator for KGE models.
  
  The evaluator encapsulates the computation of evaluation metrics based on head and tail scores. To this end, it
  offers two methods to process a batch of triples together with the scores produced by some model. It maintains
  intermediate results in its state, and offers a method to obtain the final results once finished.
class_docstr_tok: ["An","abstract","evaluator","for","KGE","models",".","The","evaluator","encapsulates","the","computation","of","evaluation","metrics","based","on","head","and","tail","scores",".","To","this","end",",","it","offers","two","methods","to","process","a","batch","of","triples","together","with","the","scores","produced","by","some","model",".","It","maintains","intermediate","results","in","its","state",",","and","offers","a","method","to","obtain","the","final","results","once","finished","."]
docstr: Evaluate metrics for model on mapped triples.
  
  :param model:
      The model to evaluate.
  :param mapped_triples:
      The triples on which to evaluate. The mapped triples should *never* contain inverse triples - these are
      created by the model class on the fly.
  :param batch_size: >0
      A positive integer used as batch size. Generally chosen as large as possible. Defaults to 1 if None.
  :param slice_size: >0
      The divisor for the scoring function when using slicing.
  :param device:
      The device on which the evaluation shall be run. If None is given, use the model's device.
  :param use_tqdm:
      Should a progress bar be displayed?
  :param tqdm_kwargs:
      Additional keyword based arguments passed to the progress bar.
  :param restrict_entities_to:
      Optionally restrict the evaluation to the given entity IDs. This may be useful if one is only interested in
      a part of the entities, e.g. due to type constraints, but wants to train on all available data. For ranking
      the entities, we still compute all scores for all possible replacement entities to avoid irregular access
      patterns which might decrease performance, but the scores will afterward be filtered to only keep those of
      interest. If provided, we assume by default that the triples are already filtered, such that it only
      contains the entities of interest. To explicitly filter within this method, pass
      `pre_filtered_triples=False`.
  :param restrict_relations_to:
      Optionally restrict the evaluation to the given relation IDs. This may be useful if one is only interested
      in a part of the relations, e.g. due to relation types, but wants to train on all available data.
      If provided, we assume by default that the triples are already filtered, such that it only contains the
      relations of interest. To explicitly filter within this method, pass `pre_filtered_triples=False`.
  :param do_time_consuming_checks:
      Whether to perform some time-consuming checks on the provided arguments. Currently, this encompasses only:
      If `restrict_entities_to` or `restrict_relations_to` is not `None`, check whether the triples have been
      filtered. Disabling this option can accelerate the method. Only effective if `pre_filtered_triples` is set
      to `True`.
  :param pre_filtered_triples:
      Whether the triples have been pre-filtered to adhere to `restrict_entities_to` / `restrict_relations_to`.
      When set to `True`, and the triples have *not* been filtered, the results may be invalid. Pre-filtering the
      triples accelerates this method, and is recommended when evaluating multiple times on the same set of
      triples.
  :param additional_filter_triples:
      additional true triples to filter out during filtered evaluation.
  :param targets:
      the prediction targets
  
  :raises NotImplementedError:
      if relation prediction evaluation is requested
  :raises ValueError:
      if the pre_filtered_triples contain unwanted entities (can only be detected with the time-consuming checks).
  :raises MemoryError:
      if the evaluation fails on cpu
  
  :return:
      the evaluation results
docstr_tok: ["Evaluate","metrics","for","model","on","mapped","triples",".",":","param","model",":","The","model","to","evaluate",".",":","param","mapped_triples",":","The","triples","on","which","to","evaluate",".","The","mapped","triples","should","*","never","*","contain","inverse","triples","-","these","are","created","by","the","model","class","on","the","fly",".",":","param","batch_size",":",">","0","A","positive","integer","used","as","batch","size",".","Generally","chosen","as","large","as","possible",".","Defaults","to","1","if","None",".",":","param","slice_size",":",">","0","The","divisor","for","the","scoring","function","when","using","slicing",".",":","param","device",":","The","device","on","which","the","evaluation","shall","be","run",".","If","None","is","given",",","use","the","model","'s","device",".",":","param","use_tqdm",":","Should","a","progress","bar","be","displayed","?",":","param","tqdm_kwargs",":","Additional","keyword","based","arguments","passed","to","the","progress","bar",".",":","param","restrict_entities_to",":","Optionally","restrict","the","evaluation","to","the","given","entity","IDs",".","This","may","be","useful","if","one","is","only","interested","in","a","part","of","the","entities",",","e.g",".","due","to","type","constraints",",","but","wants","to","train","on","all","available","data",".","For","ranking","the","entities",",","we","still","compute","all","scores","for","all","possible","replacement","entities","to","avoid","irregular","access","patterns","which","might","decrease","performance",",","but","the","scores","will","afterward","be","filtered","to","only","keep","those","of","interest",".","If","provided",",","we","assume","by","default","that","the","triples","are","already","filtered",",","such","that","it","only","contains","the","entities","of","interest",".","To","explicitly","filter","within","this","method",",","pass","`","pre_filtered_triples=False","`",".",":","param","restrict_relations_to",":","Optionally","restrict","the","evaluation","to","the","given","relation","IDs",".","This","may","be","useful","if","one","is","only","interested","in","a","part","of","the","relations",",","e.g",".","due","to","relation","types",",","but","wants","to","train","on","all","available","data",".","If","provided",",","we","assume","by","default","that","the","triples","are","already","filtered",",","such","that","it","only","contains","the","relations","of","interest",".","To","explicitly","filter","within","this","method",",","pass","`","pre_filtered_triples=False","`",".",":","param","do_time_consuming_checks",":","Whether","to","perform","some","time-consuming","checks","on","the","provided","arguments",".","Currently",",","this","encompasses","only",":","If","`","restrict_entities_to","`","or","`","restrict_relations_to","`","is","not","`","None","`",",","check","whether","the","triples","have","been","filtered",".","Disabling","this","option","can","accelerate","the","method",".","Only","effective","if","`","pre_filtered_triples","`","is","set","to","`","True","`",".",":","param","pre_filtered_triples",":","Whether","the","triples","have","been","pre-filtered","to","adhere","to","`","restrict_entities_to","`","\/","`","restrict_relations_to","`",".","When","set","to","`","True","`",",","and","the","triples","have","*","not","*","been","filtered",",","the","results","may","be","invalid",".","Pre-filtering","the","triples","accelerates","this","method",",","and","is","recommended","when","evaluating","multiple","times","on","the","same","set","of","triples",".",":","param","additional_filter_triples",":","additional","true","triples","to","filter","out","during","filtered","evaluation",".",":","param","targets",":","the","prediction","targets",":","raises","NotImplementedError",":","if","relation","prediction","evaluation","is","requested",":","raises","ValueError",":","if","the","pre_filtered_triples","contain","unwanted","entities","(","can","only","be","detected","with","the","time-consuming","checks",")",".",":","raises","MemoryError",":","if","the","evaluation","fails","on","cpu",":","return",":","the","evaluation","results"]
code: def evaluate(
      self,
      model: Model,
      mapped_triples: MappedTriples,
      batch_size: Optional[int] = None,
      slice_size: Optional[int] = None,
      device: Optional[torch.device] = None,
      use_tqdm: bool = True,
      tqdm_kwargs: Optional[Mapping[str, Any]] = None,
      restrict_entities_to: Optional[Collection[int]] = None,
      restrict_relations_to: Optional[Collection[int]] = None,
      do_time_consuming_checks: bool = True,
      additional_filter_triples: Union[
          None, MappedTriples, List[MappedTriples]
      ] = None,
      pre_filtered_triples: bool = True,
      targets: Collection[Target] = (LABEL_HEAD, LABEL_TAIL),
  ) -> MetricResults[MetricKeyType]:
      """Evaluate metrics for model on mapped triples.
  
      :param model:
          The model to evaluate.
      :param mapped_triples:
          The triples on which to evaluate. The mapped triples should *never* contain inverse triples - these are
          created by the model class on the fly.
      :param batch_size: >0
          A positive integer used as batch size. Generally chosen as large as possible. Defaults to 1 if None.
      :param slice_size: >0
          The divisor for the scoring function when using slicing.
      :param device:
          The device on which the evaluation shall be run. If None is given, use the model's device.
      :param use_tqdm:
          Should a progress bar be displayed?
      :param tqdm_kwargs:
          Additional keyword based arguments passed to the progress bar.
      :param restrict_entities_to:
          Optionally restrict the evaluation to the given entity IDs. This may be useful if one is only interested in
          a part of the entities, e.g. due to type constraints, but wants to train on all available data. For ranking
          the entities, we still compute all scores for all possible replacement entities to avoid irregular access
          patterns which might decrease performance, but the scores will afterward be filtered to only keep those of
          interest. If provided, we assume by default that the triples are already filtered, such that it only
          contains the entities of interest. To explicitly filter within this method, pass
          `pre_filtered_triples=False`.
      :param restrict_relations_to:
          Optionally restrict the evaluation to the given relation IDs. This may be useful if one is only interested
          in a part of the relations, e.g. due to relation types, but wants to train on all available data.
          If provided, we assume by default that the triples are already filtered, such that it only contains the
          relations of interest. To explicitly filter within this method, pass `pre_filtered_triples=False`.
      :param do_time_consuming_checks:
          Whether to perform some time-consuming checks on the provided arguments. Currently, this encompasses only:
          If `restrict_entities_to` or `restrict_relations_to` is not `None`, check whether the triples have been
          filtered. Disabling this option can accelerate the method. Only effective if `pre_filtered_triples` is set
          to `True`.
      :param pre_filtered_triples:
          Whether the triples have been pre-filtered to adhere to `restrict_entities_to` / `restrict_relations_to`.
          When set to `True`, and the triples have *not* been filtered, the results may be invalid. Pre-filtering the
          triples accelerates this method, and is recommended when evaluating multiple times on the same set of
          triples.
      :param additional_filter_triples:
          additional true triples to filter out during filtered evaluation.
      :param targets:
          the prediction targets
  
      :raises NotImplementedError:
          if relation prediction evaluation is requested
      :raises ValueError:
          if the pre_filtered_triples contain unwanted entities (can only be detected with the time-consuming checks).
      :raises MemoryError:
          if the evaluation fails on cpu
  
      :return:
          the evaluation results
      """
      if LABEL_RELATION in targets:
          raise NotImplementedError(
              "cf. https://github.com/pykeen/pykeen/pull/728"
          )
      start = timeit.default_timer()
  
      # verify that the triples have been filtered
      if pre_filtered_triples and do_time_consuming_checks:
          if restrict_entities_to is not None:
              present_entity_ids = get_entities(triples=mapped_triples)
              unwanted = present_entity_ids.difference(
                  restrict_entities_to
              )
              if len(unwanted) > 0:
                  raise ValueError(
                      f"mapped_triples contains IDs of entities which are not contained in restrict_entities_to:"
                      f"{unwanted}. This will invalidate the evaluation results.",
                  )
          if restrict_relations_to is not None:
              present_relation_ids = get_relations(
                  triples=mapped_triples
              )
              unwanted = present_relation_ids.difference(
                  restrict_relations_to
              )
              if len(unwanted):
                  raise ValueError(
                      f"mapped_triples contains IDs of relations which are not contained in restrict_relations_to:"
                      f"{unwanted}. This will invalidate the evaluation results.",
                  )
  
      # Filter triples if necessary
      if not pre_filtered_triples and (
          restrict_entities_to is not None
          or restrict_relations_to is not None
      ):
          old_num_triples = mapped_triples.shape[0]
          mapped_triples = restrict_triples(
              mapped_triples=mapped_triples,
              entities=restrict_entities_to,
              relations=restrict_relations_to,
          )
          logger.info(
              f"keeping {format_relative_comparison(mapped_triples.shape[0], old_num_triples)} triples."
          )
  
      # Prepare for result filtering
      if self.filtered or self.requires_positive_mask:
          all_pos_triples = prepare_filter_triples(
              mapped_triples=mapped_triples,
              additional_filter_triples=additional_filter_triples,
          )
      else:
          all_pos_triples = None
      device = device or model.device
      tqdm_kwargs = dict(tqdm_kwargs or {})
      if not use_tqdm:
          tqdm_kwargs.update(dict(disable=False))
      try:
          result = self._evaluate_on_device(
              model=model,
              mapped_triples=mapped_triples,
              batch_size=batch_size,
              slice_size=slice_size,
              device=device,
              all_pos_triples=all_pos_triples,
              restrict_entities_to=restrict_entities_to,
              targets=targets,
              tqdm_kwargs=tqdm_kwargs,
          )
      except MemoryError as error:
          if device.type == "cpu":
              raise error
          logger.error(
              f"Memory error: {error}; falling back to evaluation on cpu. This will incur heavy runtime costs for "
              f"reasonably sized datasets and models."
          )
          result = self._evaluate_on_device(
              model=model,
              mapped_triples=mapped_triples,
              batch_size=batch_size,
              slice_size=slice_size,
              device=torch.device("cpu"),
              all_pos_triples=all_pos_triples,
              restrict_entities_to=restrict_entities_to,
              targets=targets,
              tqdm_kwargs=tqdm_kwargs,
          )
      stop = timeit.default_timer()
      logger.info("Evaluation took %.2fs seconds", stop - start)
      return result
  
imports: from __future__ import annotations
  import logging
  import timeit
  import warnings
  from abc import ABC, abstractmethod
  from collections import ChainMap
  from typing import Any, ClassVar, Collection, Generic, List, Mapping, NamedTuple, Optional, Tuple, Type, TypeVar, Union, cast
  import pandas
  import torch
  from torch_max_mem import maximize_memory_utilization
  from tqdm.autonotebook import tqdm
  from ..constants import COLUMN_LABELS, TARGET_TO_INDEX, TARGET_TO_KEY_LABELS
  from ..metrics.utils import Metric
  from ..models import Model
  from ..triples.triples_factory import restrict_triples
  from ..triples.utils import get_entities, get_relations
  from ..typing import LABEL_HEAD, LABEL_RELATION, LABEL_TAIL, InductiveMode, MappedTriples, Target
  from ..utils import flatten_dictionary, format_relative_comparison, normalize_string, prepare_filter_triples
label_desc: Use image node_id 6 for calling the Evaluator obj's underlying member method code with example usage: obj.evaluate(model, mapped_triples, batch_size, slice_size, device, use_tqdm, tqdm_kwargs, restrict_entities_to, restrict_relations_to, do_time_consuming_checks, additional_filter_triples, pre_filtered_triples, targets) and returns: result