class_docstr: `BOHB <https://arxiv.org/abs/1807.01774>`__ is a robust and efficient hyperparameter tuning algorithm at scale.
  BO is an abbreviation for "Bayesian Optimization" and HB is an abbreviation for "Hyperband".
  
  BOHB relies on HB (Hyperband) to determine how many configurations to evaluate with which budget,
  but it replaces the random selection of configurations at the beginning of each HB iteration
  by a model-based search (Bayesian Optimization).
  Once the desired number of configurations for the iteration is reached,
  the standard successive halving procedure is carried out using these configurations.
  It keeps track of the performance of all function evaluations g(x, b) of configurations x
  on all budgets b to use as a basis for our models in later iterations.
  Please refer to the paper :footcite:t:`falkner2018bohb` for detailed algorithm.
  
  Note that BOHB needs additional installation using the following command:
  
  .. code-block:: bash
  
      pip install nni[BOHB]
  
  Examples
  --------
  
  .. code-block::
  
      config.tuner.name = 'BOHB'
      config.tuner.class_args = {
          'optimize_mode': 'maximize',
          'min_budget': 1,
          'max_budget': 27,
          'eta': 3,
          'min_points_in_model': 7,
          'top_n_percent': 15,
          'num_samples': 64,
          'random_fraction': 0.33,
          'bandwidth_factor': 3.0,
          'min_bandwidth': 0.001
      }
  
  Parameters
  ----------
  optimize_mode: str
      Optimize mode, 'maximize' or 'minimize'.
  min_budget: float
      The smallest budget to assign to a trial job, (budget can be the number of mini-batches or epochs).
      Needs to be positive.
  max_budget: float
      The largest budget to assign to a trial job. Needs to be larger than min_budget.
      The budgets will be geometrically distributed
      :math:`a^2 + b^2 = c^2 \sim \eta^k` for :math:`k\in [0, 1, ... , num\_subsets - 1]`.
  eta: int
      In each iteration, a complete run of sequential halving is executed. In it,
      after evaluating each configuration on the same subset size, only a fraction of
      1/eta of them 'advances' to the next round.
      Must be greater or equal to 2.
  min_points_in_model: int
      Number of observations to start building a KDE. Default 'None' means dim+1;
      when the number of completed trials in this budget is equal to or larger than ``max{dim+1, min_points_in_model}``,
      BOHB will start to build a KDE model of this budget then use said KDE model to guide configuration selection.
      Needs to be positive. (dim means the number of hyperparameters in search space)
  top_n_percent: int
      Percentage (between 1 and 99, default 15) of the observations which are considered good.
      Good points and bad points are used for building KDE models.
      For example, if you have 100 observed trials and top_n_percent is 15,
      then the top 15% of points will be used for building the good points models "l(x)".
      The remaining 85% of points will be used for building the bad point models "g(x)".
  num_samples: int
      Number of samples to optimize EI (default 64).
      In this case, it will sample "num_samples" points and compare the result of l(x)/g(x).
      Then it will return the one with the maximum l(x)/g(x) value as the next configuration
      if the optimize_mode is ``maximize``. Otherwise, it returns the smallest one.
  random_fraction: float
      Fraction of purely random configurations that are sampled from the prior without the model.
  bandwidth_factor: float
      To encourage diversity, the points proposed to optimize EI are sampled
      from a 'widened' KDE where the bandwidth is multiplied by this factor (default: 3).
      It is suggested to use the default value if you are not familiar with KDE.
  min_bandwidth: float
      To keep diversity, even when all (good) samples have the same value for one of the parameters,
      a minimum bandwidth (default: 1e-3) is used instead of zero.
      It is suggested to use the default value if you are not familiar with KDE.
  config_space: str
      Directly use a .pcs file serialized by `ConfigSpace <https://automl.github.io/ConfigSpace/>` in "pcs new" format.
      In this case, search space file (if provided in config) will be ignored.
      Note that this path needs to be an absolute path. Relative path is currently not supported.
  
  Notes
  -----
  
  Below is the introduction of the BOHB process separated in two parts:
  
  **The first part HB (Hyperband).**
  BOHB follows Hyperband’s way of choosing the budgets and continue to use SuccessiveHalving.
  For more details, you can refer to the :class:`nni.algorithms.hpo.hyperband_advisor.Hyperband`
  and the `reference paper for Hyperband <https://arxiv.org/abs/1603.06560>`__.
  This procedure is summarized by the pseudocode below.
  
  .. image:: ../../img/bohb_1.png
      :scale: 80 %
      :align: center
  
  **The second part BO (Bayesian Optimization)**
  The BO part of BOHB closely resembles TPE with one major difference:
  It opted for a single multidimensional KDE compared to the hierarchy of one-dimensional KDEs used in TPE
  in order to better handle interaction effects in the input space.
  Tree Parzen Estimator(TPE): uses a KDE (kernel density estimator) to model the densities.
  
  .. image:: ../../img/bohb_2.png
      :scale: 80 %
      :align: center
  
  To fit useful KDEs, we require a minimum number of data points Nmin;
  this is set to d + 1 for our experiments, where d is the number of hyperparameters.
  To build a model as early as possible, we do not wait until Nb = \|Db\|,
  where the number of observations for budget b is large enough to satisfy q · Nb ≥ Nmin.
  Instead, after initializing with Nmin + 2 random configurations, we choose the
  best and worst configurations, respectively, to model the two densities.
  Note that it also samples a constant fraction named **random fraction** of the configurations uniformly at random.
  
  .. image:: ../../img/bohb_3.png
      :scale: 80 %
      :align: center
  
  
  .. image:: ../../img/bohb_6.jpg
      :scale: 65 %
      :align: center
  
  **The above image shows the workflow of BOHB.**
  Here set max_budget = 9, min_budget = 1, eta = 3, others as default.
  In this case, s_max = 2, so we will continuously run the {s=2, s=1, s=0, s=2, s=1, s=0, ...} cycle.
  In each stage of SuccessiveHalving (the orange box), it will pick the top 1/eta configurations and run them again with more budget,
  repeating the SuccessiveHalving stage until the end of this iteration.
  At the same time, it collects the configurations, budgets and final metrics of each trial
  and use these to build a multidimensional KDEmodel with the key "budget".
  Multidimensional KDE is used to guide the selection of configurations for the next iteration.
  The sampling procedure (using Multidimensional KDE to guide selection) is summarized by the pseudocode below.
  
  .. image:: ../../img/bohb_4.png
      :scale: 80 %
      :align: center
  
  **Here is a simple experiment which tunes MNIST with BOHB.**
  Code implementation: :githublink:`examples/trials/mnist-advisor <examples/trials/mnist-advisor>`
  The following is the experimental final results:
  
  .. image:: ../../img/bohb_5.png
      :scale: 80 %
      :align: center
  
  More experimental results can be found in the `reference paper <https://arxiv.org/abs/1807.01774>`__.
  It shows that BOHB makes good use of previous results and has a balanced trade-off in exploration and exploitation.
class_docstr_tok: ["`","BOHB","<","https",":","\/\/arxiv.org\/abs\/1807.01774",">","`","__","is","a","robust","and","efficient","hyperparameter","tuning","algorithm","at","scale",".","BO","is","an","abbreviation","for","``","Bayesian","Optimization","''","and","HB","is","an","abbreviation","for","``","Hyperband","''",".","BOHB","relies","on","HB","(","Hyperband",")","to","determine","how","many","configurations","to","evaluate","with","which","budget",",","but","it","replaces","the","random","selection","of","configurations","at","the","beginning","of","each","HB","iteration","by","a","model-based","search","(","Bayesian","Optimization",")",".","Once","the","desired","number","of","configurations","for","the","iteration","is","reached",",","the","standard","successive","halving","procedure","is","carried","out","using","these","configurations",".","It","keeps","track","of","the","performance","of","all","function","evaluations","g","(","x",",","b",")","of","configurations","x","on","all","budgets","b","to","use","as","a","basis","for","our","models","in","later","iterations",".","Please","refer","to","the","paper",":","footcite",":","t",":","`","falkner2018bohb","`","for","detailed","algorithm",".","Note","that","BOHB","needs","additional","installation","using","the","following","command",":","..","code-block",":",":","bash","pip","install","nni","[","BOHB","]","Examples","--","--","--","--","..","code-block",":",":","config.tuner.name","=","'BOHB'","config.tuner.class_args","=","{","'optimize_mode","'",":","'maximize","'",",","'min_budget","'",":","1",",","'max_budget","'",":","27",",","'eta","'",":","3",",","'min_points_in_model","'",":","7",",","'top_n_percent","'",":","15",",","'num_samples","'",":","64",",","'random_fraction","'",":","0.33",",","'bandwidth_factor","'",":","3.0",",","'min_bandwidth","'",":","0.001","}","Parameters","--","--","--","--","--","optimize_mode",":","str","Optimize","mode",",","'maximize","'","or","'minimize","'",".","min_budget",":","float","The","smallest","budget","to","assign","to","a","trial","job",",","(","budget","can","be","the","number","of","mini-batches","or","epochs",")",".","Needs","to","be","positive",".","max_budget",":","float","The","largest","budget","to","assign","to","a","trial","job",".","Needs","to","be","larger","than","min_budget",".","The","budgets","will","be","geometrically","distributed",":","math",":","`","a^2","+","b^2","=","c^2","\\sim","\\eta^k","`","for",":","math",":","`","k\\in","[","0",",","1",",","...",",","num\\_subsets","-","1","]","`",".","eta",":","int","In","each","iteration",",","a","complete","run","of","sequential","halving","is","executed",".","In","it",",","after","evaluating","each","configuration","on","the","same","subset","size",",","only","a","fraction","of","1\/eta","of","them","'advances","'","to","the","next","round",".","Must","be","greater","or","equal","to","2.","min_points_in_model",":","int","Number","of","observations","to","start","building","a","KDE",".","Default","'None","'","means","dim+1",";","when","the","number","of","completed","trials","in","this","budget","is","equal","to","or","larger","than","``","max","{","dim+1",",","min_points_in_model","}","``",",","BOHB","will","start","to","build","a","KDE","model","of","this","budget","then","use","said","KDE","model","to","guide","configuration","selection",".","Needs","to","be","positive",".","(","dim","means","the","number","of","hyperparameters","in","search","space",")","top_n_percent",":","int","Percentage","(","between","1","and","99",",","default","15",")","of","the","observations","which","are","considered","good",".","Good","points","and","bad","points","are","used","for","building","KDE","models",".","For","example",",","if","you","have","100","observed","trials","and","top_n_percent","is","15",",","then","the","top","15","%","of","points","will","be","used","for","building","the","good","points","models","``","l","(","x",")","''",".","The","remaining","85","%","of","points","will","be","used","for","building","the","bad","point","models","``","g","(","x",")","''",".","num_samples",":","int","Number","of","samples","to","optimize","EI","(","default","64",")",".","In","this","case",",","it","will","sample","``","num_samples","''","points","and","compare","the","result","of","l","(","x",")","\/g","(","x",")",".","Then","it","will","return","the","one","with","the","maximum","l","(","x",")","\/g","(","x",")","value","as","the","next","configuration","if","the","optimize_mode","is","``","maximize","``",".","Otherwise",",","it","returns","the","smallest","one",".","random_fraction",":","float","Fraction","of","purely","random","configurations","that","are","sampled","from","the","prior","without","the","model",".","bandwidth_factor",":","float","To","encourage","diversity",",","the","points","proposed","to","optimize","EI","are","sampled","from","a","'widened","'","KDE","where","the","bandwidth","is","multiplied","by","this","factor","(","default",":","3",")",".","It","is","suggested","to","use","the","default","value","if","you","are","not","familiar","with","KDE",".","min_bandwidth",":","float","To","keep","diversity",",","even","when","all","(","good",")","samples","have","the","same","value","for","one","of","the","parameters",",","a","minimum","bandwidth","(","default",":","1e-3",")","is","used","instead","of","zero",".","It","is","suggested","to","use","the","default","value","if","you","are","not","familiar","with","KDE",".","config_space",":","str","Directly","use","a",".pcs","file","serialized","by","`","ConfigSpace","<","https",":","\/\/automl.github.io\/ConfigSpace\/",">","`","in","``","pcs","new","''","format",".","In","this","case",",","search","space","file","(","if","provided","in","config",")","will","be","ignored",".","Note","that","this","path","needs","to","be","an","absolute","path",".","Relative","path","is","currently","not","supported",".","Notes","--","--","-","Below","is","the","introduction","of","the","BOHB","process","separated","in","two","parts",":","*","*","The","first","part","HB","(","Hyperband",")",".","*","*","BOHB","follows","Hyperband","\u2019","s","way","of","choosing","the","budgets","and","continue","to","use","SuccessiveHalving",".","For","more","details",",","you","can","refer","to","the",":","class",":","`","nni.algorithms.hpo.hyperband_advisor.Hyperband","`","and","the","`","reference","paper","for","Hyperband","<","https",":","\/\/arxiv.org\/abs\/1603.06560",">","`","__",".","This","procedure","is","summarized","by","the","pseudocode","below",".","..","image",":",":","..","\/","..","\/img\/bohb_1.png",":","scale",":","80","%",":","align",":","center","*","*","The","second","part","BO","(","Bayesian","Optimization",")","*","*","The","BO","part","of","BOHB","closely","resembles","TPE","with","one","major","difference",":","It","opted","for","a","single","multidimensional","KDE","compared","to","the","hierarchy","of","one-dimensional","KDEs","used","in","TPE","in","order","to","better","handle","interaction","effects","in","the","input","space",".","Tree","Parzen","Estimator","(","TPE",")",":","uses","a","KDE","(","kernel","density","estimator",")","to","model","the","densities",".","..","image",":",":","..","\/","..","\/img\/bohb_2.png",":","scale",":","80","%",":","align",":","center","To","fit","useful","KDEs",",","we","require","a","minimum","number","of","data","points","Nmin",";","this","is","set","to","d","+","1","for","our","experiments",",","where","d","is","the","number","of","hyperparameters",".","To","build","a","model","as","early","as","possible",",","we","do","not","wait","until","Nb","=","\\|Db\\|",",","where","the","number","of","observations","for","budget","b","is","large","enough","to","satisfy","q","\u00b7","Nb","\u2265","Nmin",".","Instead",",","after","initializing","with","Nmin","+","2","random","configurations",",","we","choose","the","best","and","worst","configurations",",","respectively",",","to","model","the","two","densities",".","Note","that","it","also","samples","a","constant","fraction","named","*","*","random","fraction","*","*","of","the","configurations","uniformly","at","random",".","..","image",":",":","..","\/","..","\/img\/bohb_3.png",":","scale",":","80","%",":","align",":","center","..","image",":",":","..","\/","..","\/img\/bohb_6.jpg",":","scale",":","65","%",":","align",":","center","*","*","The","above","image","shows","the","workflow","of","BOHB",".","*","*","Here","set","max_budget","=","9",",","min_budget","=","1",",","eta","=","3",",","others","as","default",".","In","this","case",",","s_max","=","2",",","so","we","will","continuously","run","the","{","s=2",",","s=1",",","s=0",",","s=2",",","s=1",",","s=0",",","...","}","cycle",".","In","each","stage","of","SuccessiveHalving","(","the","orange","box",")",",","it","will","pick","the","top","1\/eta","configurations","and","run","them","again","with","more","budget",",","repeating","the","SuccessiveHalving","stage","until","the","end","of","this","iteration",".","At","the","same","time",",","it","collects","the","configurations",",","budgets","and","final","metrics","of","each","trial","and","use","these","to","build","a","multidimensional","KDEmodel","with","the","key","``","budget","''",".","Multidimensional","KDE","is","used","to","guide","the","selection","of","configurations","for","the","next","iteration",".","The","sampling","procedure","(","using","Multidimensional","KDE","to","guide","selection",")","is","summarized","by","the","pseudocode","below",".","..","image",":",":","..","\/","..","\/img\/bohb_4.png",":","scale",":","80","%",":","align",":","center","*","*","Here","is","a","simple","experiment","which","tunes","MNIST","with","BOHB",".","*","*","Code","implementation",":",":","githublink",":","`","examples\/trials\/mnist-advisor","<","examples\/trials\/mnist-advisor",">","`","The","following","is","the","experimental","final","results",":","..","image",":",":","..","\/","..","\/img\/bohb_5.png",":","scale",":","80","%",":","align",":","center","More","experimental","results","can","be","found","in","the","`","reference","paper","<","https",":","\/\/arxiv.org\/abs\/1807.01774",">","`","__",".","It","shows","that","BOHB","makes","good","use","of","previous","results","and","has","a","balanced","trade-off","in","exploration","and","exploitation","."]
docstr: Import additional data for tuning
  
  Parameters
  ----------
  data:
      a list of dictionarys, each of which has at least two keys, 'parameter' and 'value'
  
  Raises
  ------
  AssertionError
      data doesn't have required key 'parameter' and 'value'
code: def handle_import_data(self, data):
      """Import additional data for tuning
  
      Parameters
      ----------
      data:
          a list of dictionarys, each of which has at least two keys, 'parameter' and 'value'
  
      Raises
      ------
      AssertionError
          data doesn't have required key 'parameter' and 'value'
      """
      for entry in data:
          entry["value"] = nni.load(entry["value"])
      _completed_num = 0
      for trial_info in data:
          logger.info(
              "Importing data, current processing progress %s / %s",
              _completed_num,
              len(data),
          )
          _completed_num += 1
          assert "parameter" in trial_info
          _params = trial_info["parameter"]
          assert "value" in trial_info
          _value = trial_info["value"]
          if not _value:
              logger.info(
                  "Useless trial data, value is %s, skip this trial data.",
                  _value,
              )
              continue
          _value = extract_scalar_reward(_value)
          budget_exist_flag = False
          barely_params = dict()
          for keys in _params:
              if keys == _KEY:
                  _budget = _params[keys]
                  budget_exist_flag = True
              else:
                  barely_params[keys] = _params[keys]
          if not budget_exist_flag:
              _budget = self.max_budget
              logger.info(
                  'Set "TRIAL_BUDGET" value to %s (max budget)',
                  self.max_budget,
              )
          if self.optimize_mode is OptimizeMode.Maximize:
              reward = -_value
          else:
              reward = _value
          self.cg.new_result(
              loss=reward,
              budget=_budget,
              parameters=barely_params,
              update_model=True,
          )
      logger.info("Successfully import tuning data to BOHB advisor.")
  
imports: import sys
  import math
  import logging
  from schema import Schema, Optional
  import ConfigSpace
  import ConfigSpace.hyperparameters
  from ConfigSpace.read_and_write import pcs_new
  import nni
  from nni import ClassArgsValidator
  from nni.runtime.tuner_command_channel import CommandType
  from nni.runtime.msg_dispatcher_base import MsgDispatcherBase
  from nni.utils import OptimizeMode, MetricType, extract_scalar_reward
  from nni.runtime.common import multi_phase_enabled
  from .config_generator import CG_BOHB
label_desc: Use image node_id 13 for calling the BOHB obj's underlying member method code with example usage: obj.handle_import_data(data) without return types