We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
wrong field name in ExperimentConfig, "accessor" should be "assessor"
**Environment**:
- NNI version: 2.0
- NNI mode (local|remote|pai): local
- Client OS: Ubuntu 20.10
- Server OS (for remote mode only):
- Python version: 3.8.5
- PyTorch/TensorFlow version: pytorch 1.9.0a+a62b0de
- Is conda/virtualenv/venv used?: conda 4.9.2
- Is running in Docker?: No

**Log message**:
 - nnimanager.log: None
 - dispatcher.log: None
 - nnictl stdout and stderr:
```
WARNING: Validation with V1 schema failed. Trying to convert from V2 format...
ERROR: Conversion from v2 format failed: ValueError('ExperimentConfig: Unrecognized fields assessor')
ERROR: Config in v1 format validation failed. KeyError('trial')
```

**What issue meet, what's expected?**:
In `ExperimentConfig`, field `accessor`  should be `assessor`.

**How to reproduce it?**: 
A V2 config with 'assessor'

**Additional information**:
Fix by renaming all `accessor` in `nni.experiment.config.common.py` to `assessor`.
There are some `accessor` in doc as well.

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `nni/experiment/config/common.py`
Content:
```
1 # Copyright (c) Microsoft Corporation.
2 # Licensed under the MIT license.
3 
4 from dataclasses import dataclass
5 from pathlib import Path
6 from typing import Any, Dict, List, Optional, Union
7 
8 from .base import ConfigBase, PathLike
9 from . import util
10 
11 __all__ = [
12     'ExperimentConfig',
13     'AlgorithmConfig',
14     'CustomAlgorithmConfig',
15     'TrainingServiceConfig',
16 ]
17 
18 
19 @dataclass(init=False)
20 class _AlgorithmConfig(ConfigBase):
21     name: Optional[str] = None
22     class_name: Optional[str] = None
23     code_directory: Optional[PathLike] = None
24     class_args: Optional[Dict[str, Any]] = None
25 
26     def validate(self):
27         super().validate()
28         _validate_algo(self)
29 
30 
31 @dataclass(init=False)
32 class AlgorithmConfig(_AlgorithmConfig):
33     name: str
34     class_args: Optional[Dict[str, Any]] = None
35 
36 
37 @dataclass(init=False)
38 class CustomAlgorithmConfig(_AlgorithmConfig):
39     class_name: str
40     class_directory: Optional[PathLike] = None
41     class_args: Optional[Dict[str, Any]] = None
42 
43 
44 class TrainingServiceConfig(ConfigBase):
45     platform: str
46 
47 
48 @dataclass(init=False)
49 class ExperimentConfig(ConfigBase):
50     experiment_name: Optional[str] = None
51     search_space_file: Optional[PathLike] = None
52     search_space: Any = None
53     trial_command: str
54     trial_code_directory: PathLike = '.'
55     trial_concurrency: int
56     trial_gpu_number: Optional[int] = None
57     max_experiment_duration: Optional[str] = None
58     max_trial_number: Optional[int] = None
59     nni_manager_ip: Optional[str] = None
60     use_annotation: bool = False
61     debug: bool = False
62     log_level: Optional[str] = None
63     experiment_working_directory: Optional[PathLike] = None
64     tuner_gpu_indices: Optional[Union[List[int], str]] = None
65     tuner: Optional[_AlgorithmConfig] = None
66     accessor: Optional[_AlgorithmConfig] = None
67     advisor: Optional[_AlgorithmConfig] = None
68     training_service: Union[TrainingServiceConfig, List[TrainingServiceConfig]]
69 
70     def __init__(self, training_service_platform: Optional[Union[str, List[str]]] = None, **kwargs):
71         kwargs = util.case_insensitive(kwargs)
72         if training_service_platform is not None:
73             assert 'trainingservice' not in kwargs
74             kwargs['trainingservice'] = util.training_service_config_factory(platform = training_service_platform)
75         elif isinstance(kwargs.get('trainingservice'), (dict, list)):
76             # dict means a single training service
77             # list means hybrid training service
78             kwargs['trainingservice'] = util.training_service_config_factory(config = kwargs['trainingservice'])
79         else:
80             raise RuntimeError('Unsupported Training service configuration!')
81         super().__init__(**kwargs)
82 
83     def validate(self, initialized_tuner: bool = False) -> None:
84         super().validate()
85         if initialized_tuner:
86             _validate_for_exp(self)
87         else:
88             _validate_for_nnictl(self)
89         if self.trial_gpu_number and hasattr(self.training_service, 'use_active_gpu'):
90             if self.training_service.use_active_gpu is None:
91                 raise ValueError('Please set "use_active_gpu"')
92 
93 ## End of public API ##
94 
95     @property
96     def _canonical_rules(self):
97         return _canonical_rules
98 
99     @property
100     def _validation_rules(self):
101         return _validation_rules
102 
103 
104 _canonical_rules = {
105     'search_space_file': util.canonical_path,
106     'trial_code_directory': util.canonical_path,
107     'max_experiment_duration': lambda value: f'{util.parse_time(value)}s' if value is not None else None,
108     'experiment_working_directory': util.canonical_path,
109     'tuner_gpu_indices': lambda value: [int(idx) for idx in value.split(',')] if isinstance(value, str) else value
110 }
111 
112 _validation_rules = {
113     'search_space_file': lambda value: (Path(value).is_file(), f'"{value}" does not exist or is not regular file'),
114     'trial_code_directory': lambda value: (Path(value).is_dir(), f'"{value}" does not exist or is not directory'),
115     'trial_concurrency': lambda value: value > 0,
116     'trial_gpu_number': lambda value: value >= 0,
117     'max_experiment_duration': lambda value: util.parse_time(value) > 0,
118     'max_trial_number': lambda value: value > 0,
119     'log_level': lambda value: value in ["trace", "debug", "info", "warning", "error", "fatal"],
120     'tuner_gpu_indices': lambda value: all(i >= 0 for i in value) and len(value) == len(set(value)),
121     'training_service': lambda value: (type(value) is not TrainingServiceConfig, 'cannot be abstract base class')
122 }
123 
124 def _validate_for_exp(config: ExperimentConfig) -> None:
125     # validate experiment for nni.Experiment, where tuner is already initialized outside
126     if config.use_annotation:
127         raise ValueError('ExperimentConfig: annotation is not supported in this mode')
128     if util.count(config.search_space, config.search_space_file) != 1:
129         raise ValueError('ExperimentConfig: search_space and search_space_file must be set one')
130     if util.count(config.tuner, config.accessor, config.advisor) != 0:
131         raise ValueError('ExperimentConfig: tuner, accessor, and advisor must not be set in for this mode')
132     if config.tuner_gpu_indices is not None:
133         raise ValueError('ExperimentConfig: tuner_gpu_indices is not supported in this mode')
134 
135 def _validate_for_nnictl(config: ExperimentConfig) -> None:
136     # validate experiment for normal launching approach
137     if config.use_annotation:
138         if util.count(config.search_space, config.search_space_file) != 0:
139             raise ValueError('ExperimentConfig: search_space and search_space_file must not be set with annotationn')
140     else:
141         if util.count(config.search_space, config.search_space_file) != 1:
142             raise ValueError('ExperimentConfig: search_space and search_space_file must be set one')
143     if util.count(config.tuner, config.advisor) != 1:
144         raise ValueError('ExperimentConfig: tuner and advisor must be set one')
145 
146 def _validate_algo(algo: AlgorithmConfig) -> None:
147     if algo.name is None:
148         if algo.class_name is None:
149             raise ValueError('Missing algorithm name')
150         if algo.code_directory is not None and not Path(algo.code_directory).is_dir():
151             raise ValueError(f'code_directory "{algo.code_directory}" does not exist or is not directory')
152     else:
153         if algo.class_name is not None or algo.code_directory is not None:
154             raise ValueError(f'When name is set for registered algorithm, class_name and code_directory cannot be used')
155     # TODO: verify algorithm installation and class args
156
```

Path: `nni/assessor.py`
Content:
```
1 # Copyright (c) Microsoft Corporation.
2 # Licensed under the MIT license.
3 
4 """
5 Assessor analyzes trial's intermediate results (e.g., periodically evaluated accuracy on test dataset)
6 to tell whether this trial can be early stopped or not.
7 
8 See :class:`Assessor`' specification and ``docs/en_US/assessors.rst`` for details.
9 """
10 
11 from enum import Enum
12 import logging
13 
14 from .recoverable import Recoverable
15 
16 __all__ = ['AssessResult', 'Assessor']
17 
18 _logger = logging.getLogger(__name__)
19 
20 
21 class AssessResult(Enum):
22     """
23     Enum class for :meth:`Assessor.assess_trial` return value.
24     """
25 
26     Good = True
27     """The trial works well."""
28 
29     Bad = False
30     """The trial works poorly and should be early stopped."""
31 
32 
33 class Assessor(Recoverable):
34     """
35     Assessor analyzes trial's intermediate results (e.g., periodically evaluated accuracy on test dataset)
36     to tell whether this trial can be early stopped or not.
37 
38     This is the abstract base class for all assessors.
39     Early stopping algorithms should inherit this class and override :meth:`assess_trial` method,
40     which receives intermediate results from trials and give an assessing result.
41 
42     If :meth:`assess_trial` returns :obj:`AssessResult.Bad` for a trial,
43     it hints NNI framework that the trial is likely to result in a poor final accuracy,
44     and therefore should be killed to save resource.
45 
46     If an accessor want's to be notified when a trial ends, it can also override :meth:`trial_end`.
47 
48     To write a new assessor, you can reference :class:`~nni.medianstop_assessor.MedianstopAssessor`'s code as an example.
49 
50     See Also
51     --------
52     Builtin assessors:
53     :class:`~nni.algorithms.hpo.medianstop_assessor.MedianstopAssessor`
54     :class:`~nni.algorithms.hpo.curvefitting_assessor.CurvefittingAssessor`
55     """
56 
57     def assess_trial(self, trial_job_id, trial_history):
58         """
59         Abstract method for determining whether a trial should be killed. Must override.
60 
61         The NNI framework has little guarantee on ``trial_history``.
62         This method is not guaranteed to be invoked for each time ``trial_history`` get updated.
63         It is also possible that a trial's history keeps updating after receiving a bad result.
64         And if the trial failed and retried, ``trial_history`` may be inconsistent with its previous value.
65 
66         The only guarantee is that ``trial_history`` is always growing.
67         It will not be empty and will always be longer than previous value.
68 
69         This is an example of how :meth:`assess_trial` get invoked sequentially:
70 
71         ::
72 
73             trial_job_id | trial_history   | return value
74             ------------ | --------------- | ------------
75             Trial_A      | [1.0, 2.0]      | Good
76             Trial_B      | [1.5, 1.3]      | Bad
77             Trial_B      | [1.5, 1.3, 1.9] | Good
78             Trial_A      | [0.9, 1.8, 2.3] | Good
79 
80         Parameters
81         ----------
82         trial_job_id : str
83             Unique identifier of the trial.
84         trial_history : list
85             Intermediate results of this trial. The element type is decided by trial code.
86 
87         Returns
88         -------
89         AssessResult
90             :obj:`AssessResult.Good` or :obj:`AssessResult.Bad`.
91         """
92         raise NotImplementedError('Assessor: assess_trial not implemented')
93 
94     def trial_end(self, trial_job_id, success):
95         """
96         Abstract method invoked when a trial is completed or terminated. Do nothing by default.
97 
98         Parameters
99         ----------
100         trial_job_id : str
101             Unique identifier of the trial.
102         success : bool
103             True if the trial successfully completed; False if failed or terminated.
104         """
105 
106     def load_checkpoint(self):
107         """
108         Internal API under revising, not recommended for end users.
109         """
110         checkpoin_path = self.get_checkpoint_path()
111         _logger.info('Load checkpoint ignored by assessor, checkpoint path: %s', checkpoin_path)
112 
113     def save_checkpoint(self):
114         """
115         Internal API under revising, not recommended for end users.
116         """
117         checkpoin_path = self.get_checkpoint_path()
118         _logger.info('Save checkpoint ignored by assessor, checkpoint path: %s', checkpoin_path)
119 
120     def _on_exit(self):
121         pass
122 
123     def _on_error(self):
124         pass
125
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/nni/assessor.py b/nni/assessor.py
--- a/nni/assessor.py
+++ b/nni/assessor.py
@@ -43,7 +43,7 @@
     it hints NNI framework that the trial is likely to result in a poor final accuracy,
     and therefore should be killed to save resource.
 
-    If an accessor want's to be notified when a trial ends, it can also override :meth:`trial_end`.
+    If an assessor want's to be notified when a trial ends, it can also override :meth:`trial_end`.
 
     To write a new assessor, you can reference :class:`~nni.medianstop_assessor.MedianstopAssessor`'s code as an example.
 
diff --git a/nni/experiment/config/common.py b/nni/experiment/config/common.py
--- a/nni/experiment/config/common.py
+++ b/nni/experiment/config/common.py
@@ -63,7 +63,7 @@
     experiment_working_directory: Optional[PathLike] = None
     tuner_gpu_indices: Optional[Union[List[int], str]] = None
     tuner: Optional[_AlgorithmConfig] = None
-    accessor: Optional[_AlgorithmConfig] = None
+    assessor: Optional[_AlgorithmConfig] = None
     advisor: Optional[_AlgorithmConfig] = None
     training_service: Union[TrainingServiceConfig, List[TrainingServiceConfig]]
 
@@ -127,8 +127,8 @@
         raise ValueError('ExperimentConfig: annotation is not supported in this mode')
     if util.count(config.search_space, config.search_space_file) != 1:
         raise ValueError('ExperimentConfig: search_space and search_space_file must be set one')
-    if util.count(config.tuner, config.accessor, config.advisor) != 0:
-        raise ValueError('ExperimentConfig: tuner, accessor, and advisor must not be set in for this mode')
+    if util.count(config.tuner, config.assessor, config.advisor) != 0:
+        raise ValueError('ExperimentConfig: tuner, assessor, and advisor must not be set in for this mode')
     if config.tuner_gpu_indices is not None:
         raise ValueError('ExperimentConfig: tuner_gpu_indices is not supported in this mode')

{"golden_diff": "diff --git a/nni/assessor.py b/nni/assessor.py\n--- a/nni/assessor.py\n+++ b/nni/assessor.py\n@@ -43,7 +43,7 @@\n     it hints NNI framework that the trial is likely to result in a poor final accuracy,\n     and therefore should be killed to save resource.\n \n-    If an accessor want's to be notified when a trial ends, it can also override :meth:`trial_end`.\n+    If an assessor want's to be notified when a trial ends, it can also override :meth:`trial_end`.\n \n     To write a new assessor, you can reference :class:`~nni.medianstop_assessor.MedianstopAssessor`'s code as an example.\n \ndiff --git a/nni/experiment/config/common.py b/nni/experiment/config/common.py\n--- a/nni/experiment/config/common.py\n+++ b/nni/experiment/config/common.py\n@@ -63,7 +63,7 @@\n     experiment_working_directory: Optional[PathLike] = None\n     tuner_gpu_indices: Optional[Union[List[int], str]] = None\n     tuner: Optional[_AlgorithmConfig] = None\n-    accessor: Optional[_AlgorithmConfig] = None\n+    assessor: Optional[_AlgorithmConfig] = None\n     advisor: Optional[_AlgorithmConfig] = None\n     training_service: Union[TrainingServiceConfig, List[TrainingServiceConfig]]\n \n@@ -127,8 +127,8 @@\n         raise ValueError('ExperimentConfig: annotation is not supported in this mode')\n     if util.count(config.search_space, config.search_space_file) != 1:\n         raise ValueError('ExperimentConfig: search_space and search_space_file must be set one')\n-    if util.count(config.tuner, config.accessor, config.advisor) != 0:\n-        raise ValueError('ExperimentConfig: tuner, accessor, and advisor must not be set in for this mode')\n+    if util.count(config.tuner, config.assessor, config.advisor) != 0:\n+        raise ValueError('ExperimentConfig: tuner, assessor, and advisor must not be set in for this mode')\n     if config.tuner_gpu_indices is not None:\n         raise ValueError('ExperimentConfig: tuner_gpu_indices is not supported in this mode')\n", "issue": "wrong field name in ExperimentConfig, \"accessor\" should be \"assessor\"\n**Environment**:\r\n- NNI version: 2.0\r\n- NNI mode (local|remote|pai): local\r\n- Client OS: Ubuntu 20.10\r\n- Server OS (for remote mode only):\r\n- Python version: 3.8.5\r\n- PyTorch/TensorFlow version: pytorch 1.9.0a+a62b0de\r\n- Is conda/virtualenv/venv used?: conda 4.9.2\r\n- Is running in Docker?: No\r\n\r\n**Log message**:\r\n - nnimanager.log: None\r\n - dispatcher.log: None\r\n - nnictl stdout and stderr:\r\n```\r\nWARNING: Validation with V1 schema failed. Trying to convert from V2 format...\r\nERROR: Conversion from v2 format failed: ValueError('ExperimentConfig: Unrecognized fields assessor')\r\nERROR: Config in v1 format validation failed. KeyError('trial')\r\n```\r\n\r\n**What issue meet, what's expected?**:\r\nIn `ExperimentConfig`, field `accessor`  should be `assessor`.\r\n\r\n**How to reproduce it?**: \r\nA V2 config with 'assessor'\r\n\r\n**Additional information**:\r\nFix by renaming all `accessor` in `nni.experiment.config.common.py` to `assessor`.\r\nThere are some `accessor` in doc as well.\n", "before_files": [{"content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT license.\n\nfrom dataclasses import dataclass\nfrom pathlib import Path\nfrom typing import Any, Dict, List, Optional, Union\n\nfrom .base import ConfigBase, PathLike\nfrom . import util\n\n__all__ = [\n    'ExperimentConfig',\n    'AlgorithmConfig',\n    'CustomAlgorithmConfig',\n    'TrainingServiceConfig',\n]\n\n\n@dataclass(init=False)\nclass _AlgorithmConfig(ConfigBase):\n    name: Optional[str] = None\n    class_name: Optional[str] = None\n    code_directory: Optional[PathLike] = None\n    class_args: Optional[Dict[str, Any]] = None\n\n    def validate(self):\n        super().validate()\n        _validate_algo(self)\n\n\n@dataclass(init=False)\nclass AlgorithmConfig(_AlgorithmConfig):\n    name: str\n    class_args: Optional[Dict[str, Any]] = None\n\n\n@dataclass(init=False)\nclass CustomAlgorithmConfig(_AlgorithmConfig):\n    class_name: str\n    class_directory: Optional[PathLike] = None\n    class_args: Optional[Dict[str, Any]] = None\n\n\nclass TrainingServiceConfig(ConfigBase):\n    platform: str\n\n\n@dataclass(init=False)\nclass ExperimentConfig(ConfigBase):\n    experiment_name: Optional[str] = None\n    search_space_file: Optional[PathLike] = None\n    search_space: Any = None\n    trial_command: str\n    trial_code_directory: PathLike = '.'\n    trial_concurrency: int\n    trial_gpu_number: Optional[int] = None\n    max_experiment_duration: Optional[str] = None\n    max_trial_number: Optional[int] = None\n    nni_manager_ip: Optional[str] = None\n    use_annotation: bool = False\n    debug: bool = False\n    log_level: Optional[str] = None\n    experiment_working_directory: Optional[PathLike] = None\n    tuner_gpu_indices: Optional[Union[List[int], str]] = None\n    tuner: Optional[_AlgorithmConfig] = None\n    accessor: Optional[_AlgorithmConfig] = None\n    advisor: Optional[_AlgorithmConfig] = None\n    training_service: Union[TrainingServiceConfig, List[TrainingServiceConfig]]\n\n    def __init__(self, training_service_platform: Optional[Union[str, List[str]]] = None, **kwargs):\n        kwargs = util.case_insensitive(kwargs)\n        if training_service_platform is not None:\n            assert 'trainingservice' not in kwargs\n            kwargs['trainingservice'] = util.training_service_config_factory(platform = training_service_platform)\n        elif isinstance(kwargs.get('trainingservice'), (dict, list)):\n            # dict means a single training service\n            # list means hybrid training service\n            kwargs['trainingservice'] = util.training_service_config_factory(config = kwargs['trainingservice'])\n        else:\n            raise RuntimeError('Unsupported Training service configuration!')\n        super().__init__(**kwargs)\n\n    def validate(self, initialized_tuner: bool = False) -> None:\n        super().validate()\n        if initialized_tuner:\n            _validate_for_exp(self)\n        else:\n            _validate_for_nnictl(self)\n        if self.trial_gpu_number and hasattr(self.training_service, 'use_active_gpu'):\n            if self.training_service.use_active_gpu is None:\n                raise ValueError('Please set \"use_active_gpu\"')\n\n## End of public API ##\n\n    @property\n    def _canonical_rules(self):\n        return _canonical_rules\n\n    @property\n    def _validation_rules(self):\n        return _validation_rules\n\n\n_canonical_rules = {\n    'search_space_file': util.canonical_path,\n    'trial_code_directory': util.canonical_path,\n    'max_experiment_duration': lambda value: f'{util.parse_time(value)}s' if value is not None else None,\n    'experiment_working_directory': util.canonical_path,\n    'tuner_gpu_indices': lambda value: [int(idx) for idx in value.split(',')] if isinstance(value, str) else value\n}\n\n_validation_rules = {\n    'search_space_file': lambda value: (Path(value).is_file(), f'\"{value}\" does not exist or is not regular file'),\n    'trial_code_directory': lambda value: (Path(value).is_dir(), f'\"{value}\" does not exist or is not directory'),\n    'trial_concurrency': lambda value: value > 0,\n    'trial_gpu_number': lambda value: value >= 0,\n    'max_experiment_duration': lambda value: util.parse_time(value) > 0,\n    'max_trial_number': lambda value: value > 0,\n    'log_level': lambda value: value in [\"trace\", \"debug\", \"info\", \"warning\", \"error\", \"fatal\"],\n    'tuner_gpu_indices': lambda value: all(i >= 0 for i in value) and len(value) == len(set(value)),\n    'training_service': lambda value: (type(value) is not TrainingServiceConfig, 'cannot be abstract base class')\n}\n\ndef _validate_for_exp(config: ExperimentConfig) -> None:\n    # validate experiment for nni.Experiment, where tuner is already initialized outside\n    if config.use_annotation:\n        raise ValueError('ExperimentConfig: annotation is not supported in this mode')\n    if util.count(config.search_space, config.search_space_file) != 1:\n        raise ValueError('ExperimentConfig: search_space and search_space_file must be set one')\n    if util.count(config.tuner, config.accessor, config.advisor) != 0:\n        raise ValueError('ExperimentConfig: tuner, accessor, and advisor must not be set in for this mode')\n    if config.tuner_gpu_indices is not None:\n        raise ValueError('ExperimentConfig: tuner_gpu_indices is not supported in this mode')\n\ndef _validate_for_nnictl(config: ExperimentConfig) -> None:\n    # validate experiment for normal launching approach\n    if config.use_annotation:\n        if util.count(config.search_space, config.search_space_file) != 0:\n            raise ValueError('ExperimentConfig: search_space and search_space_file must not be set with annotationn')\n    else:\n        if util.count(config.search_space, config.search_space_file) != 1:\n            raise ValueError('ExperimentConfig: search_space and search_space_file must be set one')\n    if util.count(config.tuner, config.advisor) != 1:\n        raise ValueError('ExperimentConfig: tuner and advisor must be set one')\n\ndef _validate_algo(algo: AlgorithmConfig) -> None:\n    if algo.name is None:\n        if algo.class_name is None:\n            raise ValueError('Missing algorithm name')\n        if algo.code_directory is not None and not Path(algo.code_directory).is_dir():\n            raise ValueError(f'code_directory \"{algo.code_directory}\" does not exist or is not directory')\n    else:\n        if algo.class_name is not None or algo.code_directory is not None:\n            raise ValueError(f'When name is set for registered algorithm, class_name and code_directory cannot be used')\n    # TODO: verify algorithm installation and class args\n", "path": "nni/experiment/config/common.py"}, {"content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT license.\n\n\"\"\"\nAssessor analyzes trial's intermediate results (e.g., periodically evaluated accuracy on test dataset)\nto tell whether this trial can be early stopped or not.\n\nSee :class:`Assessor`' specification and ``docs/en_US/assessors.rst`` for details.\n\"\"\"\n\nfrom enum import Enum\nimport logging\n\nfrom .recoverable import Recoverable\n\n__all__ = ['AssessResult', 'Assessor']\n\n_logger = logging.getLogger(__name__)\n\n\nclass AssessResult(Enum):\n    \"\"\"\n    Enum class for :meth:`Assessor.assess_trial` return value.\n    \"\"\"\n\n    Good = True\n    \"\"\"The trial works well.\"\"\"\n\n    Bad = False\n    \"\"\"The trial works poorly and should be early stopped.\"\"\"\n\n\nclass Assessor(Recoverable):\n    \"\"\"\n    Assessor analyzes trial's intermediate results (e.g., periodically evaluated accuracy on test dataset)\n    to tell whether this trial can be early stopped or not.\n\n    This is the abstract base class for all assessors.\n    Early stopping algorithms should inherit this class and override :meth:`assess_trial` method,\n    which receives intermediate results from trials and give an assessing result.\n\n    If :meth:`assess_trial` returns :obj:`AssessResult.Bad` for a trial,\n    it hints NNI framework that the trial is likely to result in a poor final accuracy,\n    and therefore should be killed to save resource.\n\n    If an accessor want's to be notified when a trial ends, it can also override :meth:`trial_end`.\n\n    To write a new assessor, you can reference :class:`~nni.medianstop_assessor.MedianstopAssessor`'s code as an example.\n\n    See Also\n    --------\n    Builtin assessors:\n    :class:`~nni.algorithms.hpo.medianstop_assessor.MedianstopAssessor`\n    :class:`~nni.algorithms.hpo.curvefitting_assessor.CurvefittingAssessor`\n    \"\"\"\n\n    def assess_trial(self, trial_job_id, trial_history):\n        \"\"\"\n        Abstract method for determining whether a trial should be killed. Must override.\n\n        The NNI framework has little guarantee on ``trial_history``.\n        This method is not guaranteed to be invoked for each time ``trial_history`` get updated.\n        It is also possible that a trial's history keeps updating after receiving a bad result.\n        And if the trial failed and retried, ``trial_history`` may be inconsistent with its previous value.\n\n        The only guarantee is that ``trial_history`` is always growing.\n        It will not be empty and will always be longer than previous value.\n\n        This is an example of how :meth:`assess_trial` get invoked sequentially:\n\n        ::\n\n            trial_job_id | trial_history   | return value\n            ------------ | --------------- | ------------\n            Trial_A      | [1.0, 2.0]      | Good\n            Trial_B      | [1.5, 1.3]      | Bad\n            Trial_B      | [1.5, 1.3, 1.9] | Good\n            Trial_A      | [0.9, 1.8, 2.3] | Good\n\n        Parameters\n        ----------\n        trial_job_id : str\n            Unique identifier of the trial.\n        trial_history : list\n            Intermediate results of this trial. The element type is decided by trial code.\n\n        Returns\n        -------\n        AssessResult\n            :obj:`AssessResult.Good` or :obj:`AssessResult.Bad`.\n        \"\"\"\n        raise NotImplementedError('Assessor: assess_trial not implemented')\n\n    def trial_end(self, trial_job_id, success):\n        \"\"\"\n        Abstract method invoked when a trial is completed or terminated. Do nothing by default.\n\n        Parameters\n        ----------\n        trial_job_id : str\n            Unique identifier of the trial.\n        success : bool\n            True if the trial successfully completed; False if failed or terminated.\n        \"\"\"\n\n    def load_checkpoint(self):\n        \"\"\"\n        Internal API under revising, not recommended for end users.\n        \"\"\"\n        checkpoin_path = self.get_checkpoint_path()\n        _logger.info('Load checkpoint ignored by assessor, checkpoint path: %s', checkpoin_path)\n\n    def save_checkpoint(self):\n        \"\"\"\n        Internal API under revising, not recommended for end users.\n        \"\"\"\n        checkpoin_path = self.get_checkpoint_path()\n        _logger.info('Save checkpoint ignored by assessor, checkpoint path: %s', checkpoin_path)\n\n    def _on_exit(self):\n        pass\n\n    def _on_error(self):\n        pass\n", "path": "nni/assessor.py"}], "after_files": [{"content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT license.\n\nfrom dataclasses import dataclass\nfrom pathlib import Path\nfrom typing import Any, Dict, List, Optional, Union\n\nfrom .base import ConfigBase, PathLike\nfrom . import util\n\n__all__ = [\n    'ExperimentConfig',\n    'AlgorithmConfig',\n    'CustomAlgorithmConfig',\n    'TrainingServiceConfig',\n]\n\n\n@dataclass(init=False)\nclass _AlgorithmConfig(ConfigBase):\n    name: Optional[str] = None\n    class_name: Optional[str] = None\n    code_directory: Optional[PathLike] = None\n    class_args: Optional[Dict[str, Any]] = None\n\n    def validate(self):\n        super().validate()\n        _validate_algo(self)\n\n\n@dataclass(init=False)\nclass AlgorithmConfig(_AlgorithmConfig):\n    name: str\n    class_args: Optional[Dict[str, Any]] = None\n\n\n@dataclass(init=False)\nclass CustomAlgorithmConfig(_AlgorithmConfig):\n    class_name: str\n    class_directory: Optional[PathLike] = None\n    class_args: Optional[Dict[str, Any]] = None\n\n\nclass TrainingServiceConfig(ConfigBase):\n    platform: str\n\n\n@dataclass(init=False)\nclass ExperimentConfig(ConfigBase):\n    experiment_name: Optional[str] = None\n    search_space_file: Optional[PathLike] = None\n    search_space: Any = None\n    trial_command: str\n    trial_code_directory: PathLike = '.'\n    trial_concurrency: int\n    trial_gpu_number: Optional[int] = None\n    max_experiment_duration: Optional[str] = None\n    max_trial_number: Optional[int] = None\n    nni_manager_ip: Optional[str] = None\n    use_annotation: bool = False\n    debug: bool = False\n    log_level: Optional[str] = None\n    experiment_working_directory: Optional[PathLike] = None\n    tuner_gpu_indices: Optional[Union[List[int], str]] = None\n    tuner: Optional[_AlgorithmConfig] = None\n    assessor: Optional[_AlgorithmConfig] = None\n    advisor: Optional[_AlgorithmConfig] = None\n    training_service: Union[TrainingServiceConfig, List[TrainingServiceConfig]]\n\n    def __init__(self, training_service_platform: Optional[Union[str, List[str]]] = None, **kwargs):\n        kwargs = util.case_insensitive(kwargs)\n        if training_service_platform is not None:\n            assert 'trainingservice' not in kwargs\n            kwargs['trainingservice'] = util.training_service_config_factory(platform = training_service_platform)\n        elif isinstance(kwargs.get('trainingservice'), (dict, list)):\n            # dict means a single training service\n            # list means hybrid training service\n            kwargs['trainingservice'] = util.training_service_config_factory(config = kwargs['trainingservice'])\n        else:\n            raise RuntimeError('Unsupported Training service configuration!')\n        super().__init__(**kwargs)\n\n    def validate(self, initialized_tuner: bool = False) -> None:\n        super().validate()\n        if initialized_tuner:\n            _validate_for_exp(self)\n        else:\n            _validate_for_nnictl(self)\n        if self.trial_gpu_number and hasattr(self.training_service, 'use_active_gpu'):\n            if self.training_service.use_active_gpu is None:\n                raise ValueError('Please set \"use_active_gpu\"')\n\n## End of public API ##\n\n    @property\n    def _canonical_rules(self):\n        return _canonical_rules\n\n    @property\n    def _validation_rules(self):\n        return _validation_rules\n\n\n_canonical_rules = {\n    'search_space_file': util.canonical_path,\n    'trial_code_directory': util.canonical_path,\n    'max_experiment_duration': lambda value: f'{util.parse_time(value)}s' if value is not None else None,\n    'experiment_working_directory': util.canonical_path,\n    'tuner_gpu_indices': lambda value: [int(idx) for idx in value.split(',')] if isinstance(value, str) else value\n}\n\n_validation_rules = {\n    'search_space_file': lambda value: (Path(value).is_file(), f'\"{value}\" does not exist or is not regular file'),\n    'trial_code_directory': lambda value: (Path(value).is_dir(), f'\"{value}\" does not exist or is not directory'),\n    'trial_concurrency': lambda value: value > 0,\n    'trial_gpu_number': lambda value: value >= 0,\n    'max_experiment_duration': lambda value: util.parse_time(value) > 0,\n    'max_trial_number': lambda value: value > 0,\n    'log_level': lambda value: value in [\"trace\", \"debug\", \"info\", \"warning\", \"error\", \"fatal\"],\n    'tuner_gpu_indices': lambda value: all(i >= 0 for i in value) and len(value) == len(set(value)),\n    'training_service': lambda value: (type(value) is not TrainingServiceConfig, 'cannot be abstract base class')\n}\n\ndef _validate_for_exp(config: ExperimentConfig) -> None:\n    # validate experiment for nni.Experiment, where tuner is already initialized outside\n    if config.use_annotation:\n        raise ValueError('ExperimentConfig: annotation is not supported in this mode')\n    if util.count(config.search_space, config.search_space_file) != 1:\n        raise ValueError('ExperimentConfig: search_space and search_space_file must be set one')\n    if util.count(config.tuner, config.assessor, config.advisor) != 0:\n        raise ValueError('ExperimentConfig: tuner, assessor, and advisor must not be set in for this mode')\n    if config.tuner_gpu_indices is not None:\n        raise ValueError('ExperimentConfig: tuner_gpu_indices is not supported in this mode')\n\ndef _validate_for_nnictl(config: ExperimentConfig) -> None:\n    # validate experiment for normal launching approach\n    if config.use_annotation:\n        if util.count(config.search_space, config.search_space_file) != 0:\n            raise ValueError('ExperimentConfig: search_space and search_space_file must not be set with annotationn')\n    else:\n        if util.count(config.search_space, config.search_space_file) != 1:\n            raise ValueError('ExperimentConfig: search_space and search_space_file must be set one')\n    if util.count(config.tuner, config.advisor) != 1:\n        raise ValueError('ExperimentConfig: tuner and advisor must be set one')\n\ndef _validate_algo(algo: AlgorithmConfig) -> None:\n    if algo.name is None:\n        if algo.class_name is None:\n            raise ValueError('Missing algorithm name')\n        if algo.code_directory is not None and not Path(algo.code_directory).is_dir():\n            raise ValueError(f'code_directory \"{algo.code_directory}\" does not exist or is not directory')\n    else:\n        if algo.class_name is not None or algo.code_directory is not None:\n            raise ValueError(f'When name is set for registered algorithm, class_name and code_directory cannot be used')\n    # TODO: verify algorithm installation and class args\n", "path": "nni/experiment/config/common.py"}, {"content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT license.\n\n\"\"\"\nAssessor analyzes trial's intermediate results (e.g., periodically evaluated accuracy on test dataset)\nto tell whether this trial can be early stopped or not.\n\nSee :class:`Assessor`' specification and ``docs/en_US/assessors.rst`` for details.\n\"\"\"\n\nfrom enum import Enum\nimport logging\n\nfrom .recoverable import Recoverable\n\n__all__ = ['AssessResult', 'Assessor']\n\n_logger = logging.getLogger(__name__)\n\n\nclass AssessResult(Enum):\n    \"\"\"\n    Enum class for :meth:`Assessor.assess_trial` return value.\n    \"\"\"\n\n    Good = True\n    \"\"\"The trial works well.\"\"\"\n\n    Bad = False\n    \"\"\"The trial works poorly and should be early stopped.\"\"\"\n\n\nclass Assessor(Recoverable):\n    \"\"\"\n    Assessor analyzes trial's intermediate results (e.g., periodically evaluated accuracy on test dataset)\n    to tell whether this trial can be early stopped or not.\n\n    This is the abstract base class for all assessors.\n    Early stopping algorithms should inherit this class and override :meth:`assess_trial` method,\n    which receives intermediate results from trials and give an assessing result.\n\n    If :meth:`assess_trial` returns :obj:`AssessResult.Bad` for a trial,\n    it hints NNI framework that the trial is likely to result in a poor final accuracy,\n    and therefore should be killed to save resource.\n\n    If an assessor want's to be notified when a trial ends, it can also override :meth:`trial_end`.\n\n    To write a new assessor, you can reference :class:`~nni.medianstop_assessor.MedianstopAssessor`'s code as an example.\n\n    See Also\n    --------\n    Builtin assessors:\n    :class:`~nni.algorithms.hpo.medianstop_assessor.MedianstopAssessor`\n    :class:`~nni.algorithms.hpo.curvefitting_assessor.CurvefittingAssessor`\n    \"\"\"\n\n    def assess_trial(self, trial_job_id, trial_history):\n        \"\"\"\n        Abstract method for determining whether a trial should be killed. Must override.\n\n        The NNI framework has little guarantee on ``trial_history``.\n        This method is not guaranteed to be invoked for each time ``trial_history`` get updated.\n        It is also possible that a trial's history keeps updating after receiving a bad result.\n        And if the trial failed and retried, ``trial_history`` may be inconsistent with its previous value.\n\n        The only guarantee is that ``trial_history`` is always growing.\n        It will not be empty and will always be longer than previous value.\n\n        This is an example of how :meth:`assess_trial` get invoked sequentially:\n\n        ::\n\n            trial_job_id | trial_history   | return value\n            ------------ | --------------- | ------------\n            Trial_A      | [1.0, 2.0]      | Good\n            Trial_B      | [1.5, 1.3]      | Bad\n            Trial_B      | [1.5, 1.3, 1.9] | Good\n            Trial_A      | [0.9, 1.8, 2.3] | Good\n\n        Parameters\n        ----------\n        trial_job_id : str\n            Unique identifier of the trial.\n        trial_history : list\n            Intermediate results of this trial. The element type is decided by trial code.\n\n        Returns\n        -------\n        AssessResult\n            :obj:`AssessResult.Good` or :obj:`AssessResult.Bad`.\n        \"\"\"\n        raise NotImplementedError('Assessor: assess_trial not implemented')\n\n    def trial_end(self, trial_job_id, success):\n        \"\"\"\n        Abstract method invoked when a trial is completed or terminated. Do nothing by default.\n\n        Parameters\n        ----------\n        trial_job_id : str\n            Unique identifier of the trial.\n        success : bool\n            True if the trial successfully completed; False if failed or terminated.\n        \"\"\"\n\n    def load_checkpoint(self):\n        \"\"\"\n        Internal API under revising, not recommended for end users.\n        \"\"\"\n        checkpoin_path = self.get_checkpoint_path()\n        _logger.info('Load checkpoint ignored by assessor, checkpoint path: %s', checkpoin_path)\n\n    def save_checkpoint(self):\n        \"\"\"\n        Internal API under revising, not recommended for end users.\n        \"\"\"\n        checkpoin_path = self.get_checkpoint_path()\n        _logger.info('Save checkpoint ignored by assessor, checkpoint path: %s', checkpoin_path)\n\n    def _on_exit(self):\n        pass\n\n    def _on_error(self):\n        pass\n", "path": "nni/assessor.py"}]}