class_docstr: Base class of evaluator.
  
  To users, the evaluator is to assess the quality of a model and return a score.
  When an evaluator is defined, it usually accepts a few arguments, such as
  basic runtime information (e.g., whether to use GPU), dataset used,
  as well as hyper-parameters (such as learning rate).
  These parameters can be sometimes tunable and searched by algorithms (see :class:`MutableEvaluator`).
  
  Different evaluators could have different use scenarios and requirements on the model.
  For example, :class:`~nni.nas.evaluator.pytorch.Classification` is tailored for classification models,
  and assumes the model has a ``forward`` method that takes a batch of data and returns logits.
  Evaluators might also have different assumptions, some of which are requirements of certain algorithms.
  The evaluator with the most freedom is :class:`~nni.nas.evaluator.FunctionalEvaluator`,
  but it's also incompatible with some algorithms.
  
  To developers, the evaluator is to implement *all the logics involving forward/backward of neural networks*.
  Sometimes the algorithm requires the training and searching at the same time (e.g., one-shot algos).
  In that case, although the searching part doesn't logically belong to the evaluator,
  it is still the evaluator's responsibility to implement it,
  and the search algorithms will make sure to properly manipulate the evaluator to achieve the goal.
  
  .. tip::
  
      Inside evaluator, you can use standard :doc:`NNI trial APIs </reference/hpo>` to communicate with the exploration strategy.
      Common usages include:
  
      1. Use :func:`nni.get_current_parameter` to get the current :class:`~nni.nas.space.ExecutableModelSpace`.
         Notice that :class:`~nni.nas.space.ExecutableModelSpace` is not a directly-runnable model (e.g., a PyTorch model),
         which is different from the model received in :meth:`evaluate`.
         :class:`~nni.nas.space.ExecutableModelSpace` objects are useful for debugging, as well as for some evaluators
         which need to know extra details about how the model is sampled.
      2. Use :func:`nni.report_intermediate_result` to report intermediate results.
      3. Use :func:`nni.report_final_result` to report final results.
  
      These APIs are only available when the evaluator is executed by NNI.
      We recommend using ``nni.get_current_parameter() is not None`` to check if the APIs are available before using them.
      Please AVOID using :func:`nni.get_next_parameter()` because NAS framework has already handled the logic
      of retrieving the next parameter. Incorrectly using :func:`nni.get_next_parameter()` may cause unexpected behavior.
class_docstr_tok: ["Base","class","of","evaluator",".","To","users",",","the","evaluator","is","to","assess","the","quality","of","a","model","and","return","a","score",".","When","an","evaluator","is","defined",",","it","usually","accepts","a","few","arguments",",","such","as","basic","runtime","information","(","e.g.",",","whether","to","use","GPU",")",",","dataset","used",",","as","well","as","hyper-parameters","(","such","as","learning","rate",")",".","These","parameters","can","be","sometimes","tunable","and","searched","by","algorithms","(","see",":","class",":","`","MutableEvaluator","`",")",".","Different","evaluators","could","have","different","use","scenarios","and","requirements","on","the","model",".","For","example",",",":","class",":","`","~nni.nas.evaluator.pytorch.Classification","`","is","tailored","for","classification","models",",","and","assumes","the","model","has","a","``","forward","``","method","that","takes","a","batch","of","data","and","returns","logits",".","Evaluators","might","also","have","different","assumptions",",","some","of","which","are","requirements","of","certain","algorithms",".","The","evaluator","with","the","most","freedom","is",":","class",":","`","~nni.nas.evaluator.FunctionalEvaluator","`",",","but","it","'s","also","incompatible","with","some","algorithms",".","To","developers",",","the","evaluator","is","to","implement","*","all","the","logics","involving","forward\/backward","of","neural","networks","*",".","Sometimes","the","algorithm","requires","the","training","and","searching","at","the","same","time","(","e.g.",",","one-shot","algos",")",".","In","that","case",",","although","the","searching","part","does","n't","logically","belong","to","the","evaluator",",","it","is","still","the","evaluator","'s","responsibility","to","implement","it",",","and","the","search","algorithms","will","make","sure","to","properly","manipulate","the","evaluator","to","achieve","the","goal",".","..","tip",":",":","Inside","evaluator",",","you","can","use","standard",":","doc",":","`","NNI","trial","APIs","<","\/reference\/hpo",">","`","to","communicate","with","the","exploration","strategy",".","Common","usages","include",":","1",".","Use",":","func",":","`","nni.get_current_parameter","`","to","get","the","current",":","class",":","`","~nni.nas.space.ExecutableModelSpace","`",".","Notice","that",":","class",":","`","~nni.nas.space.ExecutableModelSpace","`","is","not","a","directly-runnable","model","(","e.g.",",","a","PyTorch","model",")",",","which","is","different","from","the","model","received","in",":","meth",":","`","evaluate","`",".",":","class",":","`","~nni.nas.space.ExecutableModelSpace","`","objects","are","useful","for","debugging",",","as","well","as","for","some","evaluators","which","need","to","know","extra","details","about","how","the","model","is","sampled",".","2",".","Use",":","func",":","`","nni.report_intermediate_result","`","to","report","intermediate","results",".","3",".","Use",":","func",":","`","nni.report_final_result","`","to","report","final","results",".","These","APIs","are","only","available","when","the","evaluator","is","executed","by","NNI",".","We","recommend","using","``","nni.get_current_parameter","(",")","is","not","None","``","to","check","if","the","APIs","are","available","before","using","them",".","Please","AVOID","using",":","func",":","`","nni.get_next_parameter","(",")","`","because","NAS","framework","has","already","handled","the","logic","of","retrieving","the","next","parameter",".","Incorrectly","using",":","func",":","`","nni.get_next_parameter","(",")","`","may","cause","unexpected","behavior","."]
docstr: Subclass implements ``_dump`` for their own serialization.
docstr_tok: ["Subclass","implements","``","_dump","``","for","their","own","serialization","."]
code: def _dump(self) -> Any:
      """Subclass implements ``_dump`` for their own serialization."""
      raise NotImplementedError()
  
imports: from __future__ import annotations
  import logging
  from contextlib import contextmanager
  from typing import Any, Callable, Iterator, Iterable, Generator, TYPE_CHECKING, cast
  from typing_extensions import Literal
  import nni
  from nni.common.serializer import is_traceable, SerializableObject
  from nni.mutable import Mutable, Sample, LabeledMutable, SampleValidationError
  from nni.mutable.mutable import _mutable_equal
  from nni.runtime.trial_command_channel import TrialCommandChannel, set_default_trial_command_channel, get_default_trial_command_channel
  from nni.typehint import ParameterRecord, TrialMetric, Parameters
label_desc: Use image node_id 6 for calling the Evaluator obj's underlying member method code with example usage: obj._dump() without return types