class_docstr: This simple annealing algorithm begins by sampling from the prior,
  but tends over time to sample from points closer and closer to the best
  ones observed.
  
  In addition to the value of this algorithm as a baseline optimization
  strategy, it is a simple starting point for implementing new algorithms.
  
  # The Annealing Algorithm
  
  The annealing algorithm is to choose one of the previous trial points
  as a starting point, and then to sample each hyperparameter from a similar
  distribution to the one specified in the prior, but whose density is more
  concentrated around the trial point we selected.
  
  This algorithm is a simple variation on random search that leverages
  smoothness in the response surface.  The annealing rate is not adaptive.
  
  ## Choosing a Best Trial
  
  The algorithm formalizes the notion of "one of the best trials" by
  sampling a position from a geometric distribution whose mean is the
  `avg_best_idx` parameter.  The "best trial" is the trial thus selected
  from the set of all trials (`self.trials`).
  
  It may happen that in the process of ancestral sampling, we may find that
  the best trial at some ancestral point did not use the hyperparameter we
  need to draw.  In such a case, this algorithm will draw a new "runner up"
  best trial, and use that one as if it had been chosen as the best trial.
  
  The set of best trials, and runner-up best trials obtained during the
  process of choosing all hyperparameters is kept sorted by the validation
  loss, and at each point where the best trial does not define a
  required hyperparameter value, we actually go through all the list of
  runners-up too, before giving up and adding a new runner-up trial.
  
  
  ## Concentrating Prior Distributions
  
  To sample a hyperparameter X within a search space, we look at
  what kind of hyperparameter it is (what kind of distribution it's from)
  and the previous successful values of that hyperparameter, and make
  a new proposal for that hyperparameter independently of other
  hyperparameters (except technically any choice nodes that led us to use
  this current hyperparameter in the first place).
  
  For example, if X is a uniform-distributed hyperparameters drawn from
  `U(l, h)`, we look at the value `x` of the hyperparameter in the selected
  trial, and draw from a new uniform density `U(x - w/2, x + w/2)`, where w
  is related to the initial range, and the number of observations we have for
  X so far. If W is the initial range, and T is the number of observations
  we have, then w = W / (1 + T * shrink_coef).  If the resulting range would
  extend either below l or above h, we shift it to fit into the original
  bounds.
class_docstr_tok: ["This","simple","annealing","algorithm","begins","by","sampling","from","the","prior",",","but","tends","over","time","to","sample","from","points","closer","and","closer","to","the","best","ones","observed",".","In","addition","to","the","value","of","this","algorithm","as","a","baseline","optimization","strategy",",","it","is","a","simple","starting","point","for","implementing","new","algorithms",".","#","The","Annealing","Algorithm","The","annealing","algorithm","is","to","choose","one","of","the","previous","trial","points","as","a","starting","point",",","and","then","to","sample","each","hyperparameter","from","a","similar","distribution","to","the","one","specified","in","the","prior",",","but","whose","density","is","more","concentrated","around","the","trial","point","we","selected",".","This","algorithm","is","a","simple","variation","on","random","search","that","leverages","smoothness","in","the","response","surface",".","The","annealing","rate","is","not","adaptive",".","#","#","Choosing","a","Best","Trial","The","algorithm","formalizes","the","notion","of","``","one","of","the","best","trials","''","by","sampling","a","position","from","a","geometric","distribution","whose","mean","is","the","`","avg_best_idx","`","parameter",".","The","``","best","trial","''","is","the","trial","thus","selected","from","the","set","of","all","trials","(","`","self.trials","`",")",".","It","may","happen","that","in","the","process","of","ancestral","sampling",",","we","may","find","that","the","best","trial","at","some","ancestral","point","did","not","use","the","hyperparameter","we","need","to","draw",".","In","such","a","case",",","this","algorithm","will","draw","a","new","``","runner","up","''","best","trial",",","and","use","that","one","as","if","it","had","been","chosen","as","the","best","trial",".","The","set","of","best","trials",",","and","runner-up","best","trials","obtained","during","the","process","of","choosing","all","hyperparameters","is","kept","sorted","by","the","validation","loss",",","and","at","each","point","where","the","best","trial","does","not","define","a","required","hyperparameter","value",",","we","actually","go","through","all","the","list","of","runners-up","too",",","before","giving","up","and","adding","a","new","runner-up","trial",".","#","#","Concentrating","Prior","Distributions","To","sample","a","hyperparameter","X","within","a","search","space",",","we","look","at","what","kind","of","hyperparameter","it","is","(","what","kind","of","distribution","it","'s","from",")","and","the","previous","successful","values","of","that","hyperparameter",",","and","make","a","new","proposal","for","that","hyperparameter","independently","of","other","hyperparameters","(","except","technically","any","choice","nodes","that","led","us","to","use","this","current","hyperparameter","in","the","first","place",")",".","For","example",",","if","X","is","a","uniform-distributed","hyperparameters","drawn","from","`","U","(","l",",","h",")","`",",","we","look","at","the","value","`","x","`","of","the","hyperparameter","in","the","selected","trial",",","and","draw","from","a","new","uniform","density","`","U","(","x","-","w\/2",",","x","+","w\/2",")","`",",","where","w","is","related","to","the","initial","range",",","and","the","number","of","observations","we","have","for","X","so","far",".","If","W","is","the","initial","range",",","and","T","is","the","number","of","observations","we","have",",","then","w","=","W","\/","(","1","+","T","*","shrink_coef",")",".","If","the","resulting","range","would","extend","either","below","l","or","above","h",",","we","shift","it","to","fit","into","the","original","bounds","."]
docstr: Parameters: See `hp_uniform`
docstr_tok: ["Parameters",":","See","`","hp_uniform","`"]
code: def hp_categorical(self, memo, node, label, tid, val):
      """
      Parameters: See `hp_uniform`
      """
      size = memo[node.arg["size"]]
      if size == 0:
          return []
      val1 = np.atleast_1d(val)
      p = p_orig = np.asarray(memo[node.arg["p"]])
      if p.ndim == 2:
          if len(p) not in (1, len(val1)):
              print(node)
              print(p)
              print(np.asarray(p).shape)
          assert len(p) in (1, len(val1))
      else:
          assert p.ndim == 1
          p = p[np.newaxis, :]
      upper = memo[node.arg["upper"]]
      if val1.size:
          counts = old_div(
              np.bincount(val1, minlength=upper), float(val1.size)
          )
          prior = self.shrinking(label)
      else:
          counts = np.zeros(upper)
          prior = 1.0
      new_p = (1 - prior) * counts + prior * p
      assert new_p.ndim == 2
      rval = categorical(p=new_p, rng=self.rng, size=size)
      if p_orig.ndim == 1:
          assert len(rval) == 1
          return rval[0]
      else:
          return rval
  
imports: from __future__ import print_function
  from __future__ import absolute_import
  from __future__ import division
  from builtins import zip
  from past.utils import old_div
  import logging
  import numpy
  from .pyll.stochastic import categorical, normal, lognormal, qnormal, qlognormal, uniform, loguniform, quniform, qloguniform
  from .base import miscs_to_idxs_vals
  from .algobase import SuggestAlgo, ExprEvaluator
label_desc: Use image node_id 10 for calling the AnnealingAlgo obj's underlying member method code with example usage: obj.hp_categorical(memo, node, label, tid, val) and returns: list, rval, rval