We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Add missing sampler and pruner classes to a tutorial page
<!-- Please write a clear and concise description of what content in https://optuna.readthedocs.io/ is an issue. -->

[Efficient Optimization Algorithms](https://optuna.readthedocs.io/en/latest/tutorial/10_key_features/003_efficient_optimization_algorithms.html) tutorial provides lists of samplers/pruners implemented in Optuna. However a few classes are not on the list. It would be great to mention them comprehensively from the tutorial pages for Optuna users. 


More concretely, please add the following entries to the tutorial page.

## Samplers

- `optuna.samplers.PartialFixedSampler`
- `optuna.samplers.NSGAIISampler`

## Pruners
- `optuna.pruners.PatientPruner`
- `optuna.pruners.PercentilePruner`


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `tutorial/10_key_features/003_efficient_optimization_algorithms.py`
Content:
```
1 """
2 .. _pruning:
3 
4 Efficient Optimization Algorithms
5 =================================
6 
7 Optuna enables efficient hyperparameter optimization by
8 adopting state-of-the-art algorithms for sampling hyperparameters and
9 pruning efficiently unpromising trials.
10 
11 Sampling Algorithms
12 -------------------
13 
14 Samplers basically continually narrow down the search space using the records of suggested parameter values and evaluated objective values,
15 leading to an optimal search space which giving off parameters leading to better objective values.
16 More detailed explanation of how samplers suggest parameters is in :class:`optuna.samplers.BaseSampler`.
17 
18 Optuna provides the following sampling algorithms:
19 
20 - Tree-structured Parzen Estimator algorithm implemented in :class:`optuna.samplers.TPESampler`
21 
22 - CMA-ES based algorithm implemented in :class:`optuna.samplers.CmaEsSampler`
23 
24 - Grid Search implemented in :class:`optuna.samplers.GridSampler`
25 
26 - Random Search implemented in :class:`optuna.samplers.RandomSampler`
27 
28 The default sampler is :class:`optuna.samplers.TPESampler`.
29 
30 Switching Samplers
31 ------------------
32 
33 """
34 
35 import optuna
36 
37 
38 ###################################################################################################
39 # By default, Optuna uses :class:`~optuna.samplers.TPESampler` as follows.
40 
41 study = optuna.create_study()
42 print(f"Sampler is {study.sampler.__class__.__name__}")
43 
44 ###################################################################################################
45 # If you want to use different samplers for example :class:`~optuna.samplers.RandomSampler`
46 # and :class:`~optuna.samplers.CmaEsSampler`,
47 
48 study = optuna.create_study(sampler=optuna.samplers.RandomSampler())
49 print(f"Sampler is {study.sampler.__class__.__name__}")
50 
51 study = optuna.create_study(sampler=optuna.samplers.CmaEsSampler())
52 print(f"Sampler is {study.sampler.__class__.__name__}")
53 
54 
55 ###################################################################################################
56 # Pruning Algorithms
57 # ------------------
58 #
59 # ``Pruners`` automatically stop unpromising trials at the early stages of the training (a.k.a., automated early-stopping).
60 #
61 # Optuna provides the following pruning algorithms:
62 #
63 # - Asynchronous Successive Halving algorithm implemented in :class:`optuna.pruners.SuccessiveHalvingPruner`
64 #
65 # - Hyperband algorithm implemented in :class:`optuna.pruners.HyperbandPruner`
66 #
67 # - Median pruning algorithm implemented in :class:`optuna.pruners.MedianPruner`
68 #
69 # - Threshold pruning algorithm implemented in :class:`optuna.pruners.ThresholdPruner`
70 #
71 # We use :class:`optuna.pruners.MedianPruner` in most examples,
72 # though basically it is outperformed by :class:`optuna.pruners.SuccessiveHalvingPruner` and
73 # :class:`optuna.pruners.HyperbandPruner` as in `this benchmark result <https://github.com/optuna/optuna/wiki/Benchmarks-with-Kurobako>`_.
74 #
75 #
76 # Activating Pruners
77 # ------------------
78 # To turn on the pruning feature, you need to call :func:`~optuna.trial.Trial.report` and :func:`~optuna.trial.Trial.should_prune` after each step of the iterative training.
79 # :func:`~optuna.trial.Trial.report` periodically monitors the intermediate objective values.
80 # :func:`~optuna.trial.Trial.should_prune` decides termination of the trial that does not meet a predefined condition.
81 #
82 # We would recommend using integration modules for major machine learning frameworks.
83 # Exclusive list is :mod:`optuna.integration` and usecases are available in  `optuna/examples <https://github.com/optuna/optuna-examples/>`_.
84 
85 
86 import logging
87 import sys
88 
89 import sklearn.datasets
90 import sklearn.linear_model
91 import sklearn.model_selection
92 
93 
94 def objective(trial):
95     iris = sklearn.datasets.load_iris()
96     classes = list(set(iris.target))
97     train_x, valid_x, train_y, valid_y = sklearn.model_selection.train_test_split(
98         iris.data, iris.target, test_size=0.25, random_state=0
99     )
100 
101     alpha = trial.suggest_float("alpha", 1e-5, 1e-1, log=True)
102     clf = sklearn.linear_model.SGDClassifier(alpha=alpha)
103 
104     for step in range(100):
105         clf.partial_fit(train_x, train_y, classes=classes)
106 
107         # Report intermediate objective value.
108         intermediate_value = 1.0 - clf.score(valid_x, valid_y)
109         trial.report(intermediate_value, step)
110 
111         # Handle pruning based on the intermediate value.
112         if trial.should_prune():
113             raise optuna.TrialPruned()
114 
115     return 1.0 - clf.score(valid_x, valid_y)
116 
117 
118 ###################################################################################################
119 # Set up the median stopping rule as the pruning condition.
120 
121 # Add stream handler of stdout to show the messages
122 optuna.logging.get_logger("optuna").addHandler(logging.StreamHandler(sys.stdout))
123 study = optuna.create_study(pruner=optuna.pruners.MedianPruner())
124 study.optimize(objective, n_trials=20)
125 
126 ###################################################################################################
127 # As you can see, several trials were pruned (stopped) before they finished all of the iterations.
128 # The format of message is ``"Trial <Trial Number> pruned."``.
129 
130 ###################################################################################################
131 # Which Sampler and Pruner Should be Used?
132 # ----------------------------------------
133 #
134 # From the benchmark results which are available at `optuna/optuna - wiki "Benchmarks with Kurobako" <https://github.com/optuna/optuna/wiki/Benchmarks-with-Kurobako>`_, at least for not deep learning tasks, we would say that
135 #
136 # * For :class:`optuna.samplers.RandomSampler`, :class:`optuna.pruners.MedianPruner` is the best.
137 # * For :class:`optuna.samplers.TPESampler`, :class:`optuna.pruners.Hyperband` is the best.
138 #
139 # However, note that the benchmark is not deep learning.
140 # For deep learning tasks,
141 # consult the below table.
142 # This table is from the `Ozaki et al., Hyperparameter Optimization Methods: Overview and Characteristics, in IEICE Trans, Vol.J103-D No.9 pp.615-631, 2020 <https://doi.org/10.14923/transinfj.2019JDR0003>`_ paper,
143 # which is written in Japanese.
144 #
145 # +---------------------------+-----------------------------------------+---------------------------------------------------------------+
146 # | Parallel Compute Resource | Categorical/Conditional Hyperparameters | Recommended Algorithms                                        |
147 # +===========================+=========================================+===============================================================+
148 # | Limited                   | No                                      | TPE. GP-EI if search space is low-dimensional and continuous. |
149 # +                           +-----------------------------------------+---------------------------------------------------------------+
150 # |                           | Yes                                     | TPE. GP-EI if search space is low-dimensional and continuous  |
151 # +---------------------------+-----------------------------------------+---------------------------------------------------------------+
152 # | Sufficient                | No                                      | CMA-ES, Random Search                                         |
153 # +                           +-----------------------------------------+---------------------------------------------------------------+
154 # |                           | Yes                                     | Random Search or Genetic Algorithm                            |
155 # +---------------------------+-----------------------------------------+---------------------------------------------------------------+
156 #
157 
158 ###################################################################################################
159 # Integration Modules for Pruning
160 # -------------------------------
161 # To implement pruning mechanism in much simpler forms, Optuna provides integration modules for the following libraries.
162 #
163 # For the complete list of Optuna's integration modules, see :mod:`optuna.integration`.
164 #
165 # For example, :class:`~optuna.integration.XGBoostPruningCallback` introduces pruning without directly changing the logic of training iteration.
166 # (See also `example <https://github.com/optuna/optuna-examples/tree/main/xgboost/xgboost_integration.py>`_ for the entire script.)
167 #
168 # .. code-block:: python
169 #
170 #         pruning_callback = optuna.integration.XGBoostPruningCallback(trial, 'validation-error')
171 #         bst = xgb.train(param, dtrain, evals=[(dvalid, 'validation')], callbacks=[pruning_callback])
172
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/tutorial/10_key_features/003_efficient_optimization_algorithms.py b/tutorial/10_key_features/003_efficient_optimization_algorithms.py
--- a/tutorial/10_key_features/003_efficient_optimization_algorithms.py
+++ b/tutorial/10_key_features/003_efficient_optimization_algorithms.py
@@ -17,13 +17,17 @@
 
 Optuna provides the following sampling algorithms:
 
+- Grid Search implemented in :class:`optuna.samplers.GridSampler`
+
+- Random Search implemented in :class:`optuna.samplers.RandomSampler`
+
 - Tree-structured Parzen Estimator algorithm implemented in :class:`optuna.samplers.TPESampler`
 
 - CMA-ES based algorithm implemented in :class:`optuna.samplers.CmaEsSampler`
 
-- Grid Search implemented in :class:`optuna.samplers.GridSampler`
+- Algorithm to enable partial fixed parameters implemented in :class:`optuna.samplers.PartialFixedSampler`
 
-- Random Search implemented in :class:`optuna.samplers.RandomSampler`
+- Nondominated Sorting Genetic Algorithm II implemented in :class:`optuna.samplers.NSGAIISampler`
 
 The default sampler is :class:`optuna.samplers.TPESampler`.
 
@@ -60,12 +64,18 @@
 #
 # Optuna provides the following pruning algorithms:
 #
+# - Median pruning algorithm implemented in :class:`optuna.pruners.MedianPruner`
+#
+# - Non-pruning algorithm implementd in :class:`optuna.pruners.NopPruner`
+#
+# - Algorithm to operate pruner with tolerance implemented in :class:`optuna.pruners.PatientPruner`
+#
+# - Algorithm to prune specified percentile of trials implemented in :class:`optuna.pruners.PercentilePruner`
+#
 # - Asynchronous Successive Halving algorithm implemented in :class:`optuna.pruners.SuccessiveHalvingPruner`
 #
 # - Hyperband algorithm implemented in :class:`optuna.pruners.HyperbandPruner`
 #
-# - Median pruning algorithm implemented in :class:`optuna.pruners.MedianPruner`
-#
 # - Threshold pruning algorithm implemented in :class:`optuna.pruners.ThresholdPruner`
 #
 # We use :class:`optuna.pruners.MedianPruner` in most examples,

{"golden_diff": "diff --git a/tutorial/10_key_features/003_efficient_optimization_algorithms.py b/tutorial/10_key_features/003_efficient_optimization_algorithms.py\n--- a/tutorial/10_key_features/003_efficient_optimization_algorithms.py\n+++ b/tutorial/10_key_features/003_efficient_optimization_algorithms.py\n@@ -17,13 +17,17 @@\n \n Optuna provides the following sampling algorithms:\n \n+- Grid Search implemented in :class:`optuna.samplers.GridSampler`\n+\n+- Random Search implemented in :class:`optuna.samplers.RandomSampler`\n+\n - Tree-structured Parzen Estimator algorithm implemented in :class:`optuna.samplers.TPESampler`\n \n - CMA-ES based algorithm implemented in :class:`optuna.samplers.CmaEsSampler`\n \n-- Grid Search implemented in :class:`optuna.samplers.GridSampler`\n+- Algorithm to enable partial fixed parameters implemented in :class:`optuna.samplers.PartialFixedSampler`\n \n-- Random Search implemented in :class:`optuna.samplers.RandomSampler`\n+- Nondominated Sorting Genetic Algorithm II implemented in :class:`optuna.samplers.NSGAIISampler`\n \n The default sampler is :class:`optuna.samplers.TPESampler`.\n \n@@ -60,12 +64,18 @@\n #\n # Optuna provides the following pruning algorithms:\n #\n+# - Median pruning algorithm implemented in :class:`optuna.pruners.MedianPruner`\n+#\n+# - Non-pruning algorithm implementd in :class:`optuna.pruners.NopPruner`\n+#\n+# - Algorithm to operate pruner with tolerance implemented in :class:`optuna.pruners.PatientPruner`\n+#\n+# - Algorithm to prune specified percentile of trials implemented in :class:`optuna.pruners.PercentilePruner`\n+#\n # - Asynchronous Successive Halving algorithm implemented in :class:`optuna.pruners.SuccessiveHalvingPruner`\n #\n # - Hyperband algorithm implemented in :class:`optuna.pruners.HyperbandPruner`\n #\n-# - Median pruning algorithm implemented in :class:`optuna.pruners.MedianPruner`\n-#\n # - Threshold pruning algorithm implemented in :class:`optuna.pruners.ThresholdPruner`\n #\n # We use :class:`optuna.pruners.MedianPruner` in most examples,\n", "issue": "Add missing sampler and pruner classes to a tutorial page\n<!-- Please write a clear and concise description of what content in https://optuna.readthedocs.io/ is an issue. -->\r\n\r\n[Efficient Optimization Algorithms](https://optuna.readthedocs.io/en/latest/tutorial/10_key_features/003_efficient_optimization_algorithms.html) tutorial provides lists of samplers/pruners implemented in Optuna. However a few classes are not on the list. It would be great to mention them comprehensively from the tutorial pages for Optuna users. \r\n\r\n\r\nMore concretely, please add the following entries to the tutorial page.\r\n\r\n## Samplers\r\n\r\n- `optuna.samplers.PartialFixedSampler`\r\n- `optuna.samplers.NSGAIISampler`\r\n\r\n## Pruners\r\n- `optuna.pruners.PatientPruner`\r\n- `optuna.pruners.PercentilePruner`\r\n\n", "before_files": [{"content": "\"\"\"\n.. _pruning:\n\nEfficient Optimization Algorithms\n=================================\n\nOptuna enables efficient hyperparameter optimization by\nadopting state-of-the-art algorithms for sampling hyperparameters and\npruning efficiently unpromising trials.\n\nSampling Algorithms\n-------------------\n\nSamplers basically continually narrow down the search space using the records of suggested parameter values and evaluated objective values,\nleading to an optimal search space which giving off parameters leading to better objective values.\nMore detailed explanation of how samplers suggest parameters is in :class:`optuna.samplers.BaseSampler`.\n\nOptuna provides the following sampling algorithms:\n\n- Tree-structured Parzen Estimator algorithm implemented in :class:`optuna.samplers.TPESampler`\n\n- CMA-ES based algorithm implemented in :class:`optuna.samplers.CmaEsSampler`\n\n- Grid Search implemented in :class:`optuna.samplers.GridSampler`\n\n- Random Search implemented in :class:`optuna.samplers.RandomSampler`\n\nThe default sampler is :class:`optuna.samplers.TPESampler`.\n\nSwitching Samplers\n------------------\n\n\"\"\"\n\nimport optuna\n\n\n###################################################################################################\n# By default, Optuna uses :class:`~optuna.samplers.TPESampler` as follows.\n\nstudy = optuna.create_study()\nprint(f\"Sampler is {study.sampler.__class__.__name__}\")\n\n###################################################################################################\n# If you want to use different samplers for example :class:`~optuna.samplers.RandomSampler`\n# and :class:`~optuna.samplers.CmaEsSampler`,\n\nstudy = optuna.create_study(sampler=optuna.samplers.RandomSampler())\nprint(f\"Sampler is {study.sampler.__class__.__name__}\")\n\nstudy = optuna.create_study(sampler=optuna.samplers.CmaEsSampler())\nprint(f\"Sampler is {study.sampler.__class__.__name__}\")\n\n\n###################################################################################################\n# Pruning Algorithms\n# ------------------\n#\n# ``Pruners`` automatically stop unpromising trials at the early stages of the training (a.k.a., automated early-stopping).\n#\n# Optuna provides the following pruning algorithms:\n#\n# - Asynchronous Successive Halving algorithm implemented in :class:`optuna.pruners.SuccessiveHalvingPruner`\n#\n# - Hyperband algorithm implemented in :class:`optuna.pruners.HyperbandPruner`\n#\n# - Median pruning algorithm implemented in :class:`optuna.pruners.MedianPruner`\n#\n# - Threshold pruning algorithm implemented in :class:`optuna.pruners.ThresholdPruner`\n#\n# We use :class:`optuna.pruners.MedianPruner` in most examples,\n# though basically it is outperformed by :class:`optuna.pruners.SuccessiveHalvingPruner` and\n# :class:`optuna.pruners.HyperbandPruner` as in `this benchmark result <https://github.com/optuna/optuna/wiki/Benchmarks-with-Kurobako>`_.\n#\n#\n# Activating Pruners\n# ------------------\n# To turn on the pruning feature, you need to call :func:`~optuna.trial.Trial.report` and :func:`~optuna.trial.Trial.should_prune` after each step of the iterative training.\n# :func:`~optuna.trial.Trial.report` periodically monitors the intermediate objective values.\n# :func:`~optuna.trial.Trial.should_prune` decides termination of the trial that does not meet a predefined condition.\n#\n# We would recommend using integration modules for major machine learning frameworks.\n# Exclusive list is :mod:`optuna.integration` and usecases are available in  `optuna/examples <https://github.com/optuna/optuna-examples/>`_.\n\n\nimport logging\nimport sys\n\nimport sklearn.datasets\nimport sklearn.linear_model\nimport sklearn.model_selection\n\n\ndef objective(trial):\n    iris = sklearn.datasets.load_iris()\n    classes = list(set(iris.target))\n    train_x, valid_x, train_y, valid_y = sklearn.model_selection.train_test_split(\n        iris.data, iris.target, test_size=0.25, random_state=0\n    )\n\n    alpha = trial.suggest_float(\"alpha\", 1e-5, 1e-1, log=True)\n    clf = sklearn.linear_model.SGDClassifier(alpha=alpha)\n\n    for step in range(100):\n        clf.partial_fit(train_x, train_y, classes=classes)\n\n        # Report intermediate objective value.\n        intermediate_value = 1.0 - clf.score(valid_x, valid_y)\n        trial.report(intermediate_value, step)\n\n        # Handle pruning based on the intermediate value.\n        if trial.should_prune():\n            raise optuna.TrialPruned()\n\n    return 1.0 - clf.score(valid_x, valid_y)\n\n\n###################################################################################################\n# Set up the median stopping rule as the pruning condition.\n\n# Add stream handler of stdout to show the messages\noptuna.logging.get_logger(\"optuna\").addHandler(logging.StreamHandler(sys.stdout))\nstudy = optuna.create_study(pruner=optuna.pruners.MedianPruner())\nstudy.optimize(objective, n_trials=20)\n\n###################################################################################################\n# As you can see, several trials were pruned (stopped) before they finished all of the iterations.\n# The format of message is ``\"Trial <Trial Number> pruned.\"``.\n\n###################################################################################################\n# Which Sampler and Pruner Should be Used?\n# ----------------------------------------\n#\n# From the benchmark results which are available at `optuna/optuna - wiki \"Benchmarks with Kurobako\" <https://github.com/optuna/optuna/wiki/Benchmarks-with-Kurobako>`_, at least for not deep learning tasks, we would say that\n#\n# * For :class:`optuna.samplers.RandomSampler`, :class:`optuna.pruners.MedianPruner` is the best.\n# * For :class:`optuna.samplers.TPESampler`, :class:`optuna.pruners.Hyperband` is the best.\n#\n# However, note that the benchmark is not deep learning.\n# For deep learning tasks,\n# consult the below table.\n# This table is from the `Ozaki et al., Hyperparameter Optimization Methods: Overview and Characteristics, in IEICE Trans, Vol.J103-D No.9 pp.615-631, 2020 <https://doi.org/10.14923/transinfj.2019JDR0003>`_ paper,\n# which is written in Japanese.\n#\n# +---------------------------+-----------------------------------------+---------------------------------------------------------------+\n# | Parallel Compute Resource | Categorical/Conditional Hyperparameters | Recommended Algorithms                                        |\n# +===========================+=========================================+===============================================================+\n# | Limited                   | No                                      | TPE. GP-EI if search space is low-dimensional and continuous. |\n# +                           +-----------------------------------------+---------------------------------------------------------------+\n# |                           | Yes                                     | TPE. GP-EI if search space is low-dimensional and continuous  |\n# +---------------------------+-----------------------------------------+---------------------------------------------------------------+\n# | Sufficient                | No                                      | CMA-ES, Random Search                                         |\n# +                           +-----------------------------------------+---------------------------------------------------------------+\n# |                           | Yes                                     | Random Search or Genetic Algorithm                            |\n# +---------------------------+-----------------------------------------+---------------------------------------------------------------+\n#\n\n###################################################################################################\n# Integration Modules for Pruning\n# -------------------------------\n# To implement pruning mechanism in much simpler forms, Optuna provides integration modules for the following libraries.\n#\n# For the complete list of Optuna's integration modules, see :mod:`optuna.integration`.\n#\n# For example, :class:`~optuna.integration.XGBoostPruningCallback` introduces pruning without directly changing the logic of training iteration.\n# (See also `example <https://github.com/optuna/optuna-examples/tree/main/xgboost/xgboost_integration.py>`_ for the entire script.)\n#\n# .. code-block:: python\n#\n#         pruning_callback = optuna.integration.XGBoostPruningCallback(trial, 'validation-error')\n#         bst = xgb.train(param, dtrain, evals=[(dvalid, 'validation')], callbacks=[pruning_callback])\n", "path": "tutorial/10_key_features/003_efficient_optimization_algorithms.py"}], "after_files": [{"content": "\"\"\"\n.. _pruning:\n\nEfficient Optimization Algorithms\n=================================\n\nOptuna enables efficient hyperparameter optimization by\nadopting state-of-the-art algorithms for sampling hyperparameters and\npruning efficiently unpromising trials.\n\nSampling Algorithms\n-------------------\n\nSamplers basically continually narrow down the search space using the records of suggested parameter values and evaluated objective values,\nleading to an optimal search space which giving off parameters leading to better objective values.\nMore detailed explanation of how samplers suggest parameters is in :class:`optuna.samplers.BaseSampler`.\n\nOptuna provides the following sampling algorithms:\n\n- Grid Search implemented in :class:`optuna.samplers.GridSampler`\n\n- Random Search implemented in :class:`optuna.samplers.RandomSampler`\n\n- Tree-structured Parzen Estimator algorithm implemented in :class:`optuna.samplers.TPESampler`\n\n- CMA-ES based algorithm implemented in :class:`optuna.samplers.CmaEsSampler`\n\n- Algorithm to enable partial fixed parameters implemented in :class:`optuna.samplers.PartialFixedSampler`\n\n- Nondominated Sorting Genetic Algorithm II implemented in :class:`optuna.samplers.NSGAIISampler`\n\nThe default sampler is :class:`optuna.samplers.TPESampler`.\n\nSwitching Samplers\n------------------\n\n\"\"\"\n\nimport optuna\n\n\n###################################################################################################\n# By default, Optuna uses :class:`~optuna.samplers.TPESampler` as follows.\n\nstudy = optuna.create_study()\nprint(f\"Sampler is {study.sampler.__class__.__name__}\")\n\n###################################################################################################\n# If you want to use different samplers for example :class:`~optuna.samplers.RandomSampler`\n# and :class:`~optuna.samplers.CmaEsSampler`,\n\nstudy = optuna.create_study(sampler=optuna.samplers.RandomSampler())\nprint(f\"Sampler is {study.sampler.__class__.__name__}\")\n\nstudy = optuna.create_study(sampler=optuna.samplers.CmaEsSampler())\nprint(f\"Sampler is {study.sampler.__class__.__name__}\")\n\n\n###################################################################################################\n# Pruning Algorithms\n# ------------------\n#\n# ``Pruners`` automatically stop unpromising trials at the early stages of the training (a.k.a., automated early-stopping).\n#\n# Optuna provides the following pruning algorithms:\n#\n# - Median pruning algorithm implemented in :class:`optuna.pruners.MedianPruner`\n#\n# - Non-pruning algorithm implementd in :class:`optuna.pruners.NopPruner`\n#\n# - Algorithm to operate pruner with tolerance implemented in :class:`optuna.pruners.PatientPruner`\n#\n# - Algorithm to prune specified percentile of trials implemented in :class:`optuna.pruners.PercentilePruner`\n#\n# - Asynchronous Successive Halving algorithm implemented in :class:`optuna.pruners.SuccessiveHalvingPruner`\n#\n# - Hyperband algorithm implemented in :class:`optuna.pruners.HyperbandPruner`\n#\n# - Threshold pruning algorithm implemented in :class:`optuna.pruners.ThresholdPruner`\n#\n# We use :class:`optuna.pruners.MedianPruner` in most examples,\n# though basically it is outperformed by :class:`optuna.pruners.SuccessiveHalvingPruner` and\n# :class:`optuna.pruners.HyperbandPruner` as in `this benchmark result <https://github.com/optuna/optuna/wiki/Benchmarks-with-Kurobako>`_.\n#\n#\n# Activating Pruners\n# ------------------\n# To turn on the pruning feature, you need to call :func:`~optuna.trial.Trial.report` and :func:`~optuna.trial.Trial.should_prune` after each step of the iterative training.\n# :func:`~optuna.trial.Trial.report` periodically monitors the intermediate objective values.\n# :func:`~optuna.trial.Trial.should_prune` decides termination of the trial that does not meet a predefined condition.\n#\n# We would recommend using integration modules for major machine learning frameworks.\n# Exclusive list is :mod:`optuna.integration` and usecases are available in  `optuna/examples <https://github.com/optuna/optuna-examples/>`_.\n\n\nimport logging\nimport sys\n\nimport sklearn.datasets\nimport sklearn.linear_model\nimport sklearn.model_selection\n\n\ndef objective(trial):\n    iris = sklearn.datasets.load_iris()\n    classes = list(set(iris.target))\n    train_x, valid_x, train_y, valid_y = sklearn.model_selection.train_test_split(\n        iris.data, iris.target, test_size=0.25, random_state=0\n    )\n\n    alpha = trial.suggest_float(\"alpha\", 1e-5, 1e-1, log=True)\n    clf = sklearn.linear_model.SGDClassifier(alpha=alpha)\n\n    for step in range(100):\n        clf.partial_fit(train_x, train_y, classes=classes)\n\n        # Report intermediate objective value.\n        intermediate_value = 1.0 - clf.score(valid_x, valid_y)\n        trial.report(intermediate_value, step)\n\n        # Handle pruning based on the intermediate value.\n        if trial.should_prune():\n            raise optuna.TrialPruned()\n\n    return 1.0 - clf.score(valid_x, valid_y)\n\n\n###################################################################################################\n# Set up the median stopping rule as the pruning condition.\n\n# Add stream handler of stdout to show the messages\noptuna.logging.get_logger(\"optuna\").addHandler(logging.StreamHandler(sys.stdout))\nstudy = optuna.create_study(pruner=optuna.pruners.MedianPruner())\nstudy.optimize(objective, n_trials=20)\n\n###################################################################################################\n# As you can see, several trials were pruned (stopped) before they finished all of the iterations.\n# The format of message is ``\"Trial <Trial Number> pruned.\"``.\n\n###################################################################################################\n# Which Sampler and Pruner Should be Used?\n# ----------------------------------------\n#\n# From the benchmark results which are available at `optuna/optuna - wiki \"Benchmarks with Kurobako\" <https://github.com/optuna/optuna/wiki/Benchmarks-with-Kurobako>`_, at least for not deep learning tasks, we would say that\n#\n# * For :class:`optuna.samplers.RandomSampler`, :class:`optuna.pruners.MedianPruner` is the best.\n# * For :class:`optuna.samplers.TPESampler`, :class:`optuna.pruners.Hyperband` is the best.\n#\n# However, note that the benchmark is not deep learning.\n# For deep learning tasks,\n# consult the below table.\n# This table is from the `Ozaki et al., Hyperparameter Optimization Methods: Overview and Characteristics, in IEICE Trans, Vol.J103-D No.9 pp.615-631, 2020 <https://doi.org/10.14923/transinfj.2019JDR0003>`_ paper,\n# which is written in Japanese.\n#\n# +---------------------------+-----------------------------------------+---------------------------------------------------------------+\n# | Parallel Compute Resource | Categorical/Conditional Hyperparameters | Recommended Algorithms                                        |\n# +===========================+=========================================+===============================================================+\n# | Limited                   | No                                      | TPE. GP-EI if search space is low-dimensional and continuous. |\n# +                           +-----------------------------------------+---------------------------------------------------------------+\n# |                           | Yes                                     | TPE. GP-EI if search space is low-dimensional and continuous  |\n# +---------------------------+-----------------------------------------+---------------------------------------------------------------+\n# | Sufficient                | No                                      | CMA-ES, Random Search                                         |\n# +                           +-----------------------------------------+---------------------------------------------------------------+\n# |                           | Yes                                     | Random Search or Genetic Algorithm                            |\n# +---------------------------+-----------------------------------------+---------------------------------------------------------------+\n#\n\n###################################################################################################\n# Integration Modules for Pruning\n# -------------------------------\n# To implement pruning mechanism in much simpler forms, Optuna provides integration modules for the following libraries.\n#\n# For the complete list of Optuna's integration modules, see :mod:`optuna.integration`.\n#\n# For example, :class:`~optuna.integration.XGBoostPruningCallback` introduces pruning without directly changing the logic of training iteration.\n# (See also `example <https://github.com/optuna/optuna-examples/tree/main/xgboost/xgboost_integration.py>`_ for the entire script.)\n#\n# .. code-block:: python\n#\n#         pruning_callback = optuna.integration.XGBoostPruningCallback(trial, 'validation-error')\n#         bst = xgb.train(param, dtrain, evals=[(dvalid, 'validation')], callbacks=[pruning_callback])\n", "path": "tutorial/10_key_features/003_efficient_optimization_algorithms.py"}]}