class_docstr: Implements a Monte Carlo tree search algorithm for reinforcement learning.
  
  This is adapted from Silver et al, "Mastering the game of Go without human
  knowledge" (https://www.nature.com/articles/nature24270).  The methods
  described in that paper rely on features of Go that are not generally true of
  all reinforcement learning problems.  To transform it into a more generally
  useful RL algorithm, it has been necessary to change some aspects of the
  method.  The overall approach used in this implementation is still the same,
  although some of the details differ.
  
  This class requires the policy to output two quantities: a vector giving the
  probability of taking each action, and an estimate of the value function for
  the current state.  At every step of simulating an episode, it performs an
  expensive tree search to explore the consequences of many possible actions.
  Based on that search, it computes much better estimates for the value function
  of the current state and the desired action probabilities.  In then tries to
  optimize the policy to make its outputs match the result of the tree search.
  
  Optimization proceeds through a series of iterations.  Each iteration consists
  of two stages:
  
  1. Simulate many episodes.  At every step perform a tree search to determine
     targets for the probabilities and value function, and store them into a
     buffer.
  2. Optimize the policy using batches drawn from the buffer generated in step 1.
  
  The tree search involves repeatedly selecting actions starting from the
  current state.  This is done by using deepcopy() to clone the environment.  It
  is essential that this produce a deterministic sequence of states: performing
  an action on the cloned environment must always lead to the same state as
  performing that action on the original environment.  For environments whose
  state transitions are deterministic, this is not a problem.  For ones whose
  state transitions are stochastic, it is essential that the random number
  generator used to select new states be stored as part of the environment and
  be properly cloned by deepcopy().
  
  This class does not support policies that include recurrent layers.
class_docstr_tok: ["Implements","a","Monte","Carlo","tree","search","algorithm","for","reinforcement","learning",".","This","is","adapted","from","Silver","et","al",",","``","Mastering","the","game","of","Go","without","human","knowledge","''","(","https",":","\/\/www.nature.com\/articles\/nature24270",")",".","The","methods","described","in","that","paper","rely","on","features","of","Go","that","are","not","generally","true","of","all","reinforcement","learning","problems",".","To","transform","it","into","a","more","generally","useful","RL","algorithm",",","it","has","been","necessary","to","change","some","aspects","of","the","method",".","The","overall","approach","used","in","this","implementation","is","still","the","same",",","although","some","of","the","details","differ",".","This","class","requires","the","policy","to","output","two","quantities",":","a","vector","giving","the","probability","of","taking","each","action",",","and","an","estimate","of","the","value","function","for","the","current","state",".","At","every","step","of","simulating","an","episode",",","it","performs","an","expensive","tree","search","to","explore","the","consequences","of","many","possible","actions",".","Based","on","that","search",",","it","computes","much","better","estimates","for","the","value","function","of","the","current","state","and","the","desired","action","probabilities",".","In","then","tries","to","optimize","the","policy","to","make","its","outputs","match","the","result","of","the","tree","search",".","Optimization","proceeds","through","a","series","of","iterations",".","Each","iteration","consists","of","two","stages",":","1",".","Simulate","many","episodes",".","At","every","step","perform","a","tree","search","to","determine","targets","for","the","probabilities","and","value","function",",","and","store","them","into","a","buffer",".","2",".","Optimize","the","policy","using","batches","drawn","from","the","buffer","generated","in","step","1",".","The","tree","search","involves","repeatedly","selecting","actions","starting","from","the","current","state",".","This","is","done","by","using","deepcopy","(",")","to","clone","the","environment",".","It","is","essential","that","this","produce","a","deterministic","sequence","of","states",":","performing","an","action","on","the","cloned","environment","must","always","lead","to","the","same","state","as","performing","that","action","on","the","original","environment",".","For","environments","whose","state","transitions","are","deterministic",",","this","is","not","a","problem",".","For","ones","whose","state","transitions","are","stochastic",",","it","is","essential","that","the","random","number","generator","used","to","select","new","states","be","stored","as","part","of","the","environment","and","be","properly","cloned","by","deepcopy","(",")",".","This","class","does","not","support","policies","that","include","recurrent","layers","."]
docstr: Perform the tree search for a state.
docstr_tok: ["Perform","the","tree","search","for","a","state","."]
code: def _do_tree_search(self, root, temperature, adapt_puct):
      """Perform the tree search for a state."""
      # Build the tree.
  
      for i in range(self.n_search_episodes):
          env = copy.deepcopy(self._env)
          self._create_trace(env, root, 1)
  
      # Compute the final probabilities and expected reward.
  
      prob = np.array(
          [c.count ** (1.0 / temperature) for c in root.children]
      )
      prob /= np.sum(prob)
      reward = np.sum(
          p * c.mean_reward for p, c in zip(prob, root.children)
      )
      if adapt_puct:
          scale = np.sum(
              [
                  p * np.abs(c.mean_reward)
                  for p, c in zip(prob, root.children)
              ]
          )
          self._puct_scale = 0.99 * self._puct_scale + 0.01 * scale
      return prob, reward
  
imports: from deepchem.models import TensorGraph
  from deepchem.models.optimizers import Adam
  from deepchem.models.tensorgraph.layers import Feature, Weights, Label, Layer
  import numpy
  import tensorflow
  import copy
  import time
label_desc: Use image node_id 10 for calling the MCTS obj's underlying member method code with example usage: obj._do_tree_search(root, temperature, adapt_puct) and returns: prob,  reward