class_docstr: A sampler using `cmaes <https://github.com/CyberAgentAILab/cmaes>`_ as the backend.
  
  Example:
  
      Optimize a simple quadratic function by using :class:`~optuna.samplers.CmaEsSampler`.
  
      .. testcode::
  
          import optuna
  
  
          def objective(trial):
              x = trial.suggest_float("x", -1, 1)
              y = trial.suggest_int("y", -1, 1)
              return x**2 + y
  
  
          sampler = optuna.samplers.CmaEsSampler()
          study = optuna.create_study(sampler=sampler)
          study.optimize(objective, n_trials=20)
  
  Please note that this sampler does not support CategoricalDistribution.
  However, :class:`~optuna.distributions.FloatDistribution` with ``step``,
  (:func:`~optuna.trial.Trial.suggest_float`) and
  :class:`~optuna.distributions.IntDistribution` (:func:`~optuna.trial.Trial.suggest_int`)
  are supported.
  
  If your search space contains categorical parameters, I recommend you
  to use :class:`~optuna.samplers.TPESampler` instead.
  Furthermore, there is room for performance improvements in parallel
  optimization settings. This sampler cannot use some trials for updating
  the parameters of multivariate normal distribution.
  
  For further information about CMA-ES algorithm, please refer to the following papers:
  
  - `N. Hansen, The CMA Evolution Strategy: A Tutorial. arXiv:1604.00772, 2016.
    <https://arxiv.org/abs/1604.00772>`_
  - `A. Auger and N. Hansen. A restart CMA evolution strategy with increasing population
    size. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC 2005),
    pages 1769–1776. IEEE Press, 2005.
    <http://www.cmap.polytechnique.fr/~nikolaus.hansen/cec2005ipopcmaes.pdf>`_
  - `N. Hansen. Benchmarking a BI-Population CMA-ES on the BBOB-2009 Function Testbed.
    GECCO Workshop, 2009.
    <https://dl.acm.org/doi/10.1145/1570256.1570333>`_
  - `Raymond Ros, Nikolaus Hansen. A Simple Modification in CMA-ES Achieving Linear Time and
    Space Complexity. 10th International Conference on Parallel Problem Solving From Nature,
    Sep 2008, Dortmund, Germany. inria-00287367.
    <https://hal.inria.fr/inria-00287367/document>`_
  - `Masahiro Nomura, Shuhei Watanabe, Youhei Akimoto, Yoshihiko Ozaki, Masaki Onishi.
    Warm Starting CMA-ES for Hyperparameter Optimization, AAAI. 2021.
    <https://arxiv.org/abs/2012.06932>`_
  - `R. Hamano, S. Saito, M. Nomura, S. Shirakawa. CMA-ES with Margin: Lower-Bounding Marginal
    Probability for Mixed-Integer Black-Box Optimization, GECCO. 2022.
    <https://arxiv.org/abs/2205.13482>`_
  - `M. Nomura, Y. Akimoto, I. Ono. CMA-ES with Learning Rate Adaptation: Can CMA-ES with
    Default Population Size Solve Multimodal and Noisy Problems?, GECCO. 2023.
    <https://arxiv.org/abs/2304.03473>`_
  
  .. seealso::
      You can also use :class:`optuna.integration.PyCmaSampler` which is a sampler using cma
      library as the backend.
  
  Args:
  
      x0:
          A dictionary of an initial parameter values for CMA-ES. By default, the mean of ``low``
          and ``high`` for each distribution is used. Note that ``x0`` is sampled uniformly
          within the search space domain for each restart if you specify ``restart_strategy``
          argument.
  
      sigma0:
          Initial standard deviation of CMA-ES. By default, ``sigma0`` is set to
          ``min_range / 6``, where ``min_range`` denotes the minimum range of the distributions
          in the search space.
  
      seed:
          A random seed for CMA-ES.
  
      n_startup_trials:
          The independent sampling is used instead of the CMA-ES algorithm until the given number
          of trials finish in the same study.
  
      independent_sampler:
          A :class:`~optuna.samplers.BaseSampler` instance that is used for independent
          sampling. The parameters not contained in the relative search space are sampled
          by this sampler.
          The search space for :class:`~optuna.samplers.CmaEsSampler` is determined by
          :func:`~optuna.search_space.intersection_search_space()`.
  
          If :obj:`None` is specified, :class:`~optuna.samplers.RandomSampler` is used
          as the default.
  
          .. seealso::
              :class:`optuna.samplers` module provides built-in independent samplers
              such as :class:`~optuna.samplers.RandomSampler` and
              :class:`~optuna.samplers.TPESampler`.
  
      warn_independent_sampling:
          If this is :obj:`True`, a warning message is emitted when
          the value of a parameter is sampled by using an independent sampler.
  
          Note that the parameters of the first trial in a study are always sampled
          via an independent sampler, so no warning messages are emitted in this case.
  
      restart_strategy:
          Strategy for restarting CMA-ES optimization when converges to a local minimum.
          If :obj:`None` is given, CMA-ES will not restart (default).
          If 'ipop' is given, CMA-ES will restart with increasing population size.
          if 'bipop' is given, CMA-ES will restart with the population size
          increased or decreased.
          Please see also ``inc_popsize`` parameter.
  
          .. note::
              Added in v2.1.0 as an experimental feature. The interface may change in newer
              versions without prior notice. See
              https://github.com/optuna/optuna/releases/tag/v2.1.0.
  
      popsize:
          A population size of CMA-ES. When ``restart_strategy = 'ipop'``
          or ``restart_strategy = 'bipop'`` is specified,
          this is used as the initial population size.
  
      inc_popsize:
          Multiplier for increasing population size before each restart.
          This argument will be used when ``restart_strategy = 'ipop'``
          or ``restart_strategy = 'bipop'`` is specified.
  
      consider_pruned_trials:
          If this is :obj:`True`, the PRUNED trials are considered for sampling.
  
          .. note::
              Added in v2.0.0 as an experimental feature. The interface may change in newer
              versions without prior notice. See
              https://github.com/optuna/optuna/releases/tag/v2.0.0.
  
          .. note::
              It is suggested to set this flag :obj:`False` when the
              :class:`~optuna.pruners.MedianPruner` is used. On the other hand, it is suggested
              to set this flag :obj:`True` when the :class:`~optuna.pruners.HyperbandPruner` is
              used. Please see `the benchmark result
              <https://github.com/optuna/optuna/pull/1229>`_ for the details.
  
      use_separable_cma:
          If this is :obj:`True`, the covariance matrix is constrained to be diagonal.
          Due to reduce the model complexity, the learning rate for the covariance matrix
          is increased. Consequently, this algorithm outperforms CMA-ES on separable functions.
  
          .. note::
              Added in v2.6.0 as an experimental feature. The interface may change in newer
              versions without prior notice. See
              https://github.com/optuna/optuna/releases/tag/v2.6.0.
  
      with_margin:
          If this is :obj:`True`, CMA-ES with margin is used. This algorithm prevents samples in
          each discrete distribution (:class:`~optuna.distributions.FloatDistribution` with
          `step` and :class:`~optuna.distributions.IntDistribution`) from being fixed to a single
          point.
          Currently, this option cannot be used with ``use_separable_cma=True``.
  
          .. note::
              Added in v3.1.0 as an experimental feature. The interface may change in newer
              versions without prior notice. See
              https://github.com/optuna/optuna/releases/tag/v3.1.0.
  
      lr_adapt:
          If this is :obj:`True`, CMA-ES with learning rate adaptation is used.
          This algorithm focuses on working well on multimodal and/or noisy problems
          with default settings.
          Currently, this option cannot be used with ``use_separable_cma=True`` or
          ``with_margin=True``.
  
          .. note::
              Added in v3.3.0 or later, as an experimental feature.
              The interface may change in newer versions without prior notice. See
              https://github.com/optuna/optuna/releases/tag/v3.3.0.
  
      source_trials:
          This option is for Warm Starting CMA-ES, a method to transfer prior knowledge on
          similar HPO tasks through the initialization of CMA-ES. This method estimates a
          promising distribution from ``source_trials`` and generates the parameter of
          multivariate gaussian distribution. Please note that it is prohibited to use
          ``x0``, ``sigma0``, or ``use_separable_cma`` argument together.
  
          .. note::
              Added in v2.6.0 as an experimental feature. The interface may change in newer
              versions without prior notice. See
              https://github.com/optuna/optuna/releases/tag/v2.6.0.
class_docstr_tok: ["A","sampler","using","`","cmaes","<","https",":","\/\/github.com\/CyberAgentAILab\/cmaes",">","`","_","as","the","backend",".","Example",":","Optimize","a","simple","quadratic","function","by","using",":","class",":","`","~optuna.samplers.CmaEsSampler","`",".","..","testcode",":",":","import","optuna","def","objective","(","trial",")",":","x","=","trial.suggest_float","(","``","x","''",",","-1",",","1",")","y","=","trial.suggest_int","(","``","y","''",",","-1",",","1",")","return","x","*","*","2","+","y","sampler","=","optuna.samplers.CmaEsSampler","(",")","study","=","optuna.create_study","(","sampler=sampler",")","study.optimize","(","objective",",","n_trials=20",")","Please","note","that","this","sampler","does","not","support","CategoricalDistribution",".","However",",",":","class",":","`","~optuna.distributions.FloatDistribution","`","with","``","step","``",",","(",":","func",":","`","~optuna.trial.Trial.suggest_float","`",")","and",":","class",":","`","~optuna.distributions.IntDistribution","`","(",":","func",":","`","~optuna.trial.Trial.suggest_int","`",")","are","supported",".","If","your","search","space","contains","categorical","parameters",",","I","recommend","you","to","use",":","class",":","`","~optuna.samplers.TPESampler","`","instead",".","Furthermore",",","there","is","room","for","performance","improvements","in","parallel","optimization","settings",".","This","sampler","can","not","use","some","trials","for","updating","the","parameters","of","multivariate","normal","distribution",".","For","further","information","about","CMA-ES","algorithm",",","please","refer","to","the","following","papers",":","-","`","N.","Hansen",",","The","CMA","Evolution","Strategy",":","A","Tutorial",".","arXiv:1604.00772",",","2016",".","<","https",":","\/\/arxiv.org\/abs\/1604.00772",">","`","_","-","`","A.","Auger","and","N.","Hansen",".","A","restart","CMA","evolution","strategy","with","increasing","population","size",".","In","Proceedings","of","the","IEEE","Congress","on","Evolutionary","Computation","(","CEC","2005",")",",","pages","1769\u20131776",".","IEEE","Press",",","2005",".","<","http",":","\/\/www.cmap.polytechnique.fr\/~nikolaus.hansen\/cec2005ipopcmaes.pdf",">","`","_","-","`","N.","Hansen",".","Benchmarking","a","BI-Population","CMA-ES","on","the","BBOB-2009","Function","Testbed",".","GECCO","Workshop",",","2009",".","<","https",":","\/\/dl.acm.org\/doi\/10.1145\/1570256.1570333",">","`","_","-","`","Raymond","Ros",",","Nikolaus","Hansen",".","A","Simple","Modification","in","CMA-ES","Achieving","Linear","Time","and","Space","Complexity",".","10th","International","Conference","on","Parallel","Problem","Solving","From","Nature",",","Sep","2008",",","Dortmund",",","Germany",".","inria-00287367",".","<","https",":","\/\/hal.inria.fr\/inria-00287367\/document",">","`","_","-","`","Masahiro","Nomura",",","Shuhei","Watanabe",",","Youhei","Akimoto",",","Yoshihiko","Ozaki",",","Masaki","Onishi",".","Warm","Starting","CMA-ES","for","Hyperparameter","Optimization",",","AAAI",".","2021",".","<","https",":","\/\/arxiv.org\/abs\/2012.06932",">","`","_","-","`","R.","Hamano",",","S.","Saito",",","M.","Nomura",",","S.","Shirakawa",".","CMA-ES","with","Margin",":","Lower-Bounding","Marginal","Probability","for","Mixed-Integer","Black-Box","Optimization",",","GECCO",".","2022",".","<","https",":","\/\/arxiv.org\/abs\/2205.13482",">","`","_","-","`","M.","Nomura",",","Y.","Akimoto",",","I.","Ono",".","CMA-ES","with","Learning","Rate","Adaptation",":","Can","CMA-ES","with","Default","Population","Size","Solve","Multimodal","and","Noisy","Problems","?",",","GECCO",".","2023",".","<","https",":","\/\/arxiv.org\/abs\/2304.03473",">","`","_","..","seealso",":",":","You","can","also","use",":","class",":","`","optuna.integration.PyCmaSampler","`","which","is","a","sampler","using","cma","library","as","the","backend",".","Args",":","x0",":","A","dictionary","of","an","initial","parameter","values","for","CMA-ES",".","By","default",",","the","mean","of","``","low","``","and","``","high","``","for","each","distribution","is","used",".","Note","that","``","x0","``","is","sampled","uniformly","within","the","search","space","domain","for","each","restart","if","you","specify","``","restart_strategy","``","argument",".","sigma0",":","Initial","standard","deviation","of","CMA-ES",".","By","default",",","``","sigma0","``","is","set","to","``","min_range","\/","6","``",",","where","``","min_range","``","denotes","the","minimum","range","of","the","distributions","in","the","search","space",".","seed",":","A","random","seed","for","CMA-ES",".","n_startup_trials",":","The","independent","sampling","is","used","instead","of","the","CMA-ES","algorithm","until","the","given","number","of","trials","finish","in","the","same","study",".","independent_sampler",":","A",":","class",":","`","~optuna.samplers.BaseSampler","`","instance","that","is","used","for","independent","sampling",".","The","parameters","not","contained","in","the","relative","search","space","are","sampled","by","this","sampler",".","The","search","space","for",":","class",":","`","~optuna.samplers.CmaEsSampler","`","is","determined","by",":","func",":","`","~optuna.search_space.intersection_search_space","(",")","`",".","If",":","obj",":","`","None","`","is","specified",",",":","class",":","`","~optuna.samplers.RandomSampler","`","is","used","as","the","default",".","..","seealso",":",":",":","class",":","`","optuna.samplers","`","module","provides","built-in","independent","samplers","such","as",":","class",":","`","~optuna.samplers.RandomSampler","`","and",":","class",":","`","~optuna.samplers.TPESampler","`",".","warn_independent_sampling",":","If","this","is",":","obj",":","`","True","`",",","a","warning","message","is","emitted","when","the","value","of","a","parameter","is","sampled","by","using","an","independent","sampler",".","Note","that","the","parameters","of","the","first","trial","in","a","study","are","always","sampled","via","an","independent","sampler",",","so","no","warning","messages","are","emitted","in","this","case",".","restart_strategy",":","Strategy","for","restarting","CMA-ES","optimization","when","converges","to","a","local","minimum",".","If",":","obj",":","`","None","`","is","given",",","CMA-ES","will","not","restart","(","default",")",".","If","'ipop","'","is","given",",","CMA-ES","will","restart","with","increasing","population","size",".","if","'bipop","'","is","given",",","CMA-ES","will","restart","with","the","population","size","increased","or","decreased",".","Please","see","also","``","inc_popsize","``","parameter",".","..","note",":",":","Added","in","v2.1.0","as","an","experimental","feature",".","The","interface","may","change","in","newer","versions","without","prior","notice",".","See","https",":","\/\/github.com\/optuna\/optuna\/releases\/tag\/v2.1.0",".","popsize",":","A","population","size","of","CMA-ES",".","When","``","restart_strategy","=","'ipop","'","``","or","``","restart_strategy","=","'bipop","'","``","is","specified",",","this","is","used","as","the","initial","population","size",".","inc_popsize",":","Multiplier","for","increasing","population","size","before","each","restart",".","This","argument","will","be","used","when","``","restart_strategy","=","'ipop","'","``","or","``","restart_strategy","=","'bipop","'","``","is","specified",".","consider_pruned_trials",":","If","this","is",":","obj",":","`","True","`",",","the","PRUNED","trials","are","considered","for","sampling",".","..","note",":",":","Added","in","v2.0.0","as","an","experimental","feature",".","The","interface","may","change","in","newer","versions","without","prior","notice",".","See","https",":","\/\/github.com\/optuna\/optuna\/releases\/tag\/v2.0.0",".","..","note",":",":","It","is","suggested","to","set","this","flag",":","obj",":","`","False","`","when","the",":","class",":","`","~optuna.pruners.MedianPruner","`","is","used",".","On","the","other","hand",",","it","is","suggested","to","set","this","flag",":","obj",":","`","True","`","when","the",":","class",":","`","~optuna.pruners.HyperbandPruner","`","is","used",".","Please","see","`","the","benchmark","result","<","https",":","\/\/github.com\/optuna\/optuna\/pull\/1229",">","`","_","for","the","details",".","use_separable_cma",":","If","this","is",":","obj",":","`","True","`",",","the","covariance","matrix","is","constrained","to","be","diagonal",".","Due","to","reduce","the","model","complexity",",","the","learning","rate","for","the","covariance","matrix","is","increased",".","Consequently",",","this","algorithm","outperforms","CMA-ES","on","separable","functions",".","..","note",":",":","Added","in","v2.6.0","as","an","experimental","feature",".","The","interface","may","change","in","newer","versions","without","prior","notice",".","See","https",":","\/\/github.com\/optuna\/optuna\/releases\/tag\/v2.6.0",".","with_margin",":","If","this","is",":","obj",":","`","True","`",",","CMA-ES","with","margin","is","used",".","This","algorithm","prevents","samples","in","each","discrete","distribution","(",":","class",":","`","~optuna.distributions.FloatDistribution","`","with","`","step","`","and",":","class",":","`","~optuna.distributions.IntDistribution","`",")","from","being","fixed","to","a","single","point",".","Currently",",","this","option","can","not","be","used","with","``","use_separable_cma=True","``",".","..","note",":",":","Added","in","v3.1.0","as","an","experimental","feature",".","The","interface","may","change","in","newer","versions","without","prior","notice",".","See","https",":","\/\/github.com\/optuna\/optuna\/releases\/tag\/v3.1.0",".","lr_adapt",":","If","this","is",":","obj",":","`","True","`",",","CMA-ES","with","learning","rate","adaptation","is","used",".","This","algorithm","focuses","on","working","well","on","multimodal","and\/or","noisy","problems","with","default","settings",".","Currently",",","this","option","can","not","be","used","with","``","use_separable_cma=True","``","or","``","with_margin=True","``",".","..","note",":",":","Added","in","v3.3.0","or","later",",","as","an","experimental","feature",".","The","interface","may","change","in","newer","versions","without","prior","notice",".","See","https",":","\/\/github.com\/optuna\/optuna\/releases\/tag\/v3.3.0",".","source_trials",":","This","option","is","for","Warm","Starting","CMA-ES",",","a","method","to","transfer","prior","knowledge","on","similar","HPO","tasks","through","the","initialization","of","CMA-ES",".","This","method","estimates","a","promising","distribution","from","``","source_trials","``","and","generates","the","parameter","of","multivariate","gaussian","distribution",".","Please","note","that","it","is","prohibited","to","use","``","x0","``",",","``","sigma0","``",",","or","``","use_separable_cma","``","argument","together",".","..","note",":",":","Added","in","v2.6.0","as","an","experimental","feature",".","The","interface","may","change","in","newer","versions","without","prior","notice",".","See","https",":","\/\/github.com\/optuna\/optuna\/releases\/tag\/v2.6.0","."]
code: def _concat_optimizer_attrs(
      self, optimizer_attrs: Dict[str, str], n_restarts: int = 0
  ) -> str:
      return "".join(
          optimizer_attrs[
              "{}:{}".format(self._attr_keys.optimizer(n_restarts), i)
          ]
          for i in range(len(optimizer_attrs))
      )
  
imports: from __future__ import annotations
  import copy
  import math
  import pickle
  from typing import Any
  from typing import Callable
  from typing import cast
  from typing import Dict
  from typing import List
  from typing import NamedTuple
  from typing import Optional
  from typing import Sequence
  from typing import Tuple
  from typing import TYPE_CHECKING
  from typing import Union
  import warnings
  import numpy
  import optuna
  from optuna import logging
  from optuna._imports import _LazyImport
  from optuna._transform import _SearchSpaceTransform
  from optuna.distributions import BaseDistribution
  from optuna.distributions import FloatDistribution
  from optuna.distributions import IntDistribution
  from optuna.exceptions import ExperimentalWarning
  from optuna.samplers import BaseSampler
  from optuna.samplers._lazy_random_state import LazyRandomState
  from optuna.search_space import IntersectionSearchSpace
  from optuna.study._study_direction import StudyDirection
  from optuna.trial import FrozenTrial
  from optuna.trial import TrialState
label_desc: Use image node_id 6 for calling the CmaEsSampler obj's underlying member method code with example usage: obj._concat_optimizer_attrs(optimizer_attrs, n_restarts) and returns: str