class_docstr: Implements a Monte Carlo tree search algorithm for reinforcement learning.
  
  This is adapted from Silver et al, "Mastering the game of Go without human
  knowledge" (https://www.nature.com/articles/nature24270).  The methods
  described in that paper rely on features of Go that are not generally true of
  all reinforcement learning problems.  To transform it into a more generally
  useful RL algorithm, it has been necessary to change some aspects of the
  method.  The overall approach used in this implementation is still the same,
  although some of the details differ.
  
  This class requires the policy to output two quantities: a vector giving the
  probability of taking each action, and an estimate of the value function for
  the current state.  At every step of simulating an episode, it performs an
  expensive tree search to explore the consequences of many possible actions.
  Based on that search, it computes much better estimates for the value function
  of the current state and the desired action probabilities.  In then tries to
  optimize the policy to make its outputs match the result of the tree search.
  
  Optimization proceeds through a series of iterations.  Each iteration consists
  of two stages:
  
  1. Simulate many episodes.  At every step perform a tree search to determine
     targets for the probabilities and value function, and store them into a
     buffer.
  2. Optimize the policy using batches drawn from the buffer generated in step 1.
  
  The tree search involves repeatedly selecting actions starting from the
  current state.  This is done by using deepcopy() to clone the environment.  It
  is essential that this produce a deterministic sequence of states: performing
  an action on the cloned environment must always lead to the same state as
  performing that action on the original environment.  For environments whose
  state transitions are deterministic, this is not a problem.  For ones whose
  state transitions are stochastic, it is essential that the random number
  generator used to select new states be stored as part of the environment and
  be properly cloned by deepcopy().
  
  This class does not support policies that include recurrent layers.
class_docstr_tok: ["Implements","a","Monte","Carlo","tree","search","algorithm","for","reinforcement","learning",".","This","is","adapted","from","Silver","et","al",",","``","Mastering","the","game","of","Go","without","human","knowledge","''","(","https",":","\/\/www.nature.com\/articles\/nature24270",")",".","The","methods","described","in","that","paper","rely","on","features","of","Go","that","are","not","generally","true","of","all","reinforcement","learning","problems",".","To","transform","it","into","a","more","generally","useful","RL","algorithm",",","it","has","been","necessary","to","change","some","aspects","of","the","method",".","The","overall","approach","used","in","this","implementation","is","still","the","same",",","although","some","of","the","details","differ",".","This","class","requires","the","policy","to","output","two","quantities",":","a","vector","giving","the","probability","of","taking","each","action",",","and","an","estimate","of","the","value","function","for","the","current","state",".","At","every","step","of","simulating","an","episode",",","it","performs","an","expensive","tree","search","to","explore","the","consequences","of","many","possible","actions",".","Based","on","that","search",",","it","computes","much","better","estimates","for","the","value","function","of","the","current","state","and","the","desired","action","probabilities",".","In","then","tries","to","optimize","the","policy","to","make","its","outputs","match","the","result","of","the","tree","search",".","Optimization","proceeds","through","a","series","of","iterations",".","Each","iteration","consists","of","two","stages",":","1",".","Simulate","many","episodes",".","At","every","step","perform","a","tree","search","to","determine","targets","for","the","probabilities","and","value","function",",","and","store","them","into","a","buffer",".","2",".","Optimize","the","policy","using","batches","drawn","from","the","buffer","generated","in","step","1",".","The","tree","search","involves","repeatedly","selecting","actions","starting","from","the","current","state",".","This","is","done","by","using","deepcopy","(",")","to","clone","the","environment",".","It","is","essential","that","this","produce","a","deterministic","sequence","of","states",":","performing","an","action","on","the","cloned","environment","must","always","lead","to","the","same","state","as","performing","that","action","on","the","original","environment",".","For","environments","whose","state","transitions","are","deterministic",",","this","is","not","a","problem",".","For","ones","whose","state","transitions","are","stochastic",",","it","is","essential","that","the","random","number","generator","used","to","select","new","states","be","stored","as","part","of","the","environment","and","be","properly","cloned","by","deepcopy","(",")",".","This","class","does","not","support","policies","that","include","recurrent","layers","."]
docstr: Create an object for optimizing a policy.
  
  Parameters
  ----------
  env: Environment
    the Environment to interact with
  policy: Policy
    the Policy to optimize.  Its create_layers() method must return a dict containing the
    keys 'action_prob' and 'value', corresponding to the action probabilities and value estimate
  max_search_depth: int
    the maximum depth of the tree search, measured in steps
  n_search_episodes: int
    the number of episodes to simulate (up to max_search_depth, if they do not
    terminate first) for each tree search
  discount_factor: float
    the discount factor to use when computing rewards
  value_weight: float
    a scale factor for the value loss term in the loss function
  optimizer: Optimizer
    the optimizer to use
  model_dir: str
    the directory in which the model will be saved.  If None, a temporary directory will be created.
docstr_tok: ["Create","an","object","for","optimizing","a","policy",".","Parameters","--","--","--","--","--","env",":","Environment","the","Environment","to","interact","with","policy",":","Policy","the","Policy","to","optimize",".","Its","create_layers","(",")","method","must","return","a","dict","containing","the","keys","'action_prob","'","and","'value","'",",","corresponding","to","the","action","probabilities","and","value","estimate","max_search_depth",":","int","the","maximum","depth","of","the","tree","search",",","measured","in","steps","n_search_episodes",":","int","the","number","of","episodes","to","simulate","(","up","to","max_search_depth",",","if","they","do","not","terminate","first",")","for","each","tree","search","discount_factor",":","float","the","discount","factor","to","use","when","computing","rewards","value_weight",":","float","a","scale","factor","for","the","value","loss","term","in","the","loss","function","optimizer",":","Optimizer","the","optimizer","to","use","model_dir",":","str","the","directory","in","which","the","model","will","be","saved",".","If","None",",","a","temporary","directory","will","be","created","."]
code: def __init__(
      self,
      env,
      policy,
      max_search_depth=100,
      n_search_episodes=1000,
      discount_factor=0.99,
      value_weight=1.0,
      optimizer=Adam(),
      model_dir=None,
  ):
      """Create an object for optimizing a policy.
  
      Parameters
      ----------
      env: Environment
        the Environment to interact with
      policy: Policy
        the Policy to optimize.  Its create_layers() method must return a dict containing the
        keys 'action_prob' and 'value', corresponding to the action probabilities and value estimate
      max_search_depth: int
        the maximum depth of the tree search, measured in steps
      n_search_episodes: int
        the number of episodes to simulate (up to max_search_depth, if they do not
        terminate first) for each tree search
      discount_factor: float
        the discount factor to use when computing rewards
      value_weight: float
        a scale factor for the value loss term in the loss function
      optimizer: Optimizer
        the optimizer to use
      model_dir: str
        the directory in which the model will be saved.  If None, a temporary directory will be created.
      """
      self._env = copy.deepcopy(env)
      self._policy = policy
      self.max_search_depth = max_search_depth
      self.n_search_episodes = n_search_episodes
      self.discount_factor = discount_factor
      self.value_weight = value_weight
      self._state_is_list = isinstance(
          env.state_shape[0], SequenceCollection
      )
      if optimizer is None:
          self._optimizer = Adam(
              learning_rate=0.001, beta1=0.9, beta2=0.999
          )
      else:
          self._optimizer = optimizer
      (
          self._graph,
          self._features,
          self._pred_prob,
          self._pred_value,
          self._search_prob,
          self._search_value,
      ) = self._build_graph(None, "global", model_dir)
      with self._graph._get_tf("Graph").as_default():
          with tf.variable_scope("global"):
              self._checkpoint = tf.train.Checkpoint()
              self._checkpoint.save_counter  # Ensure the variable has been created
          self._checkpoint.listed = tf.get_collection(
              tf.GraphKeys.GLOBAL_VARIABLES, scope="global"
          )
          self._graph.session.run(
              self._checkpoint.save_counter.initializer
          )
  
imports: from deepchem.models import TensorGraph
  from deepchem.models.optimizers import Adam
  from deepchem.models.tensorgraph.layers import Feature, Weights, Label, Layer
  import numpy
  import tensorflow
  import copy
  import time
label_desc: Use image node_id 1 to create a new MCTS object from inherited base classes: object with example: obj = MCTS(env, policy, max_search_depth, n_search_episodes, discount_factor, value_weight, optimizer, model_dir)