class_docstr: Implements a Monte Carlo tree search algorithm for reinforcement learning.
  
  This is adapted from Silver et al, "Mastering the game of Go without human
  knowledge" (https://www.nature.com/articles/nature24270).  The methods
  described in that paper rely on features of Go that are not generally true of
  all reinforcement learning problems.  To transform it into a more generally
  useful RL algorithm, it has been necessary to change some aspects of the
  method.  The overall approach used in this implementation is still the same,
  although some of the details differ.
  
  This class requires the policy to output two quantities: a vector giving the
  probability of taking each action, and an estimate of the value function for
  the current state.  At every step of simulating an episode, it performs an
  expensive tree search to explore the consequences of many possible actions.
  Based on that search, it computes much better estimates for the value function
  of the current state and the desired action probabilities.  In then tries to
  optimize the policy to make its outputs match the result of the tree search.
  
  Optimization proceeds through a series of iterations.  Each iteration consists
  of two stages:
  
  1. Simulate many episodes.  At every step perform a tree search to determine
     targets for the probabilities and value function, and store them into a
     buffer.
  2. Optimize the policy using batches drawn from the buffer generated in step 1.
  
  The tree search involves repeatedly selecting actions starting from the
  current state.  This is done by using deepcopy() to clone the environment.  It
  is essential that this produce a deterministic sequence of states: performing
  an action on the cloned environment must always lead to the same state as
  performing that action on the original environment.  For environments whose
  state transitions are deterministic, this is not a problem.  For ones whose
  state transitions are stochastic, it is essential that the random number
  generator used to select new states be stored as part of the environment and
  be properly cloned by deepcopy().
  
  This class does not support policies that include recurrent layers.
class_docstr_tok: ["Implements","a","Monte","Carlo","tree","search","algorithm","for","reinforcement","learning",".","This","is","adapted","from","Silver","et","al",",","``","Mastering","the","game","of","Go","without","human","knowledge","''","(","https",":","\/\/www.nature.com\/articles\/nature24270",")",".","The","methods","described","in","that","paper","rely","on","features","of","Go","that","are","not","generally","true","of","all","reinforcement","learning","problems",".","To","transform","it","into","a","more","generally","useful","RL","algorithm",",","it","has","been","necessary","to","change","some","aspects","of","the","method",".","The","overall","approach","used","in","this","implementation","is","still","the","same",",","although","some","of","the","details","differ",".","This","class","requires","the","policy","to","output","two","quantities",":","a","vector","giving","the","probability","of","taking","each","action",",","and","an","estimate","of","the","value","function","for","the","current","state",".","At","every","step","of","simulating","an","episode",",","it","performs","an","expensive","tree","search","to","explore","the","consequences","of","many","possible","actions",".","Based","on","that","search",",","it","computes","much","better","estimates","for","the","value","function","of","the","current","state","and","the","desired","action","probabilities",".","In","then","tries","to","optimize","the","policy","to","make","its","outputs","match","the","result","of","the","tree","search",".","Optimization","proceeds","through","a","series","of","iterations",".","Each","iteration","consists","of","two","stages",":","1",".","Simulate","many","episodes",".","At","every","step","perform","a","tree","search","to","determine","targets","for","the","probabilities","and","value","function",",","and","store","them","into","a","buffer",".","2",".","Optimize","the","policy","using","batches","drawn","from","the","buffer","generated","in","step","1",".","The","tree","search","involves","repeatedly","selecting","actions","starting","from","the","current","state",".","This","is","done","by","using","deepcopy","(",")","to","clone","the","environment",".","It","is","essential","that","this","produce","a","deterministic","sequence","of","states",":","performing","an","action","on","the","cloned","environment","must","always","lead","to","the","same","state","as","performing","that","action","on","the","original","environment",".","For","environments","whose","state","transitions","are","deterministic",",","this","is","not","a","problem",".","For","ones","whose","state","transitions","are","stochastic",",","it","is","essential","that","the","random","number","generator","used","to","select","new","states","be","stored","as","part","of","the","environment","and","be","properly","cloned","by","deepcopy","(",")",".","This","class","does","not","support","policies","that","include","recurrent","layers","."]
docstr: Train the policy.
  
  Parameters
  ----------
  iterations: int
    the total number of iterations (simulation followed by optimization) to perform
  steps_per_iteration: int
    the total number of steps to simulate in each iteration.  Every step consists
    of a tree search, followed by selecting an action based on the results of
    the search.
  epochs_per_iteration: int
    the number of epochs of optimization to perform for each iteration.  Each
    epoch involves randomly ordering all the steps that were just simulated in
    the current iteration, splitting them into batches, and looping over the
    batches.
  temperature: float
    the temperature factor to use when selecting a move for each step of
    simulation.  Larger values produce a broader probability distribution and
    hence more exploration.  Smaller values produce a stronger preference for
    whatever action did best in the tree search.
  puct_scale: float
    the scale of the PUCT term in the expression for selecting actions during
    tree search.  This should be roughly similar in magnitude to the rewards
    given by the environment, since the PUCT term is added to the mean
    discounted reward.  This may be None, in which case a value is adaptively
    selected that tries to match the mean absolute value of the discounted
    reward.
  max_checkpoints_to_keep: int
    the maximum number of checkpoint files to keep.  When this number is reached, older
    files are deleted.
  checkpoint_interval: float
    the time interval at which to save checkpoints, measured in seconds
  restore: bool
    if True, restore the model from the most recent checkpoint and continue training
    from there.  If False, retrain the model from scratch.
docstr_tok: ["Train","the","policy",".","Parameters","--","--","--","--","--","iterations",":","int","the","total","number","of","iterations","(","simulation","followed","by","optimization",")","to","perform","steps_per_iteration",":","int","the","total","number","of","steps","to","simulate","in","each","iteration",".","Every","step","consists","of","a","tree","search",",","followed","by","selecting","an","action","based","on","the","results","of","the","search",".","epochs_per_iteration",":","int","the","number","of","epochs","of","optimization","to","perform","for","each","iteration",".","Each","epoch","involves","randomly","ordering","all","the","steps","that","were","just","simulated","in","the","current","iteration",",","splitting","them","into","batches",",","and","looping","over","the","batches",".","temperature",":","float","the","temperature","factor","to","use","when","selecting","a","move","for","each","step","of","simulation",".","Larger","values","produce","a","broader","probability","distribution","and","hence","more","exploration",".","Smaller","values","produce","a","stronger","preference","for","whatever","action","did","best","in","the","tree","search",".","puct_scale",":","float","the","scale","of","the","PUCT","term","in","the","expression","for","selecting","actions","during","tree","search",".","This","should","be","roughly","similar","in","magnitude","to","the","rewards","given","by","the","environment",",","since","the","PUCT","term","is","added","to","the","mean","discounted","reward",".","This","may","be","None",",","in","which","case","a","value","is","adaptively","selected","that","tries","to","match","the","mean","absolute","value","of","the","discounted","reward",".","max_checkpoints_to_keep",":","int","the","maximum","number","of","checkpoint","files","to","keep",".","When","this","number","is","reached",",","older","files","are","deleted",".","checkpoint_interval",":","float","the","time","interval","at","which","to","save","checkpoints",",","measured","in","seconds","restore",":","bool","if","True",",","restore","the","model","from","the","most","recent","checkpoint","and","continue","training","from","there",".","If","False",",","retrain","the","model","from","scratch","."]
code: def fit(
      self,
      iterations,
      steps_per_iteration=10000,
      epochs_per_iteration=10,
      temperature=0.5,
      puct_scale=None,
      max_checkpoints_to_keep=5,
      checkpoint_interval=600,
      restore=False,
  ):
      """Train the policy.
  
      Parameters
      ----------
      iterations: int
        the total number of iterations (simulation followed by optimization) to perform
      steps_per_iteration: int
        the total number of steps to simulate in each iteration.  Every step consists
        of a tree search, followed by selecting an action based on the results of
        the search.
      epochs_per_iteration: int
        the number of epochs of optimization to perform for each iteration.  Each
        epoch involves randomly ordering all the steps that were just simulated in
        the current iteration, splitting them into batches, and looping over the
        batches.
      temperature: float
        the temperature factor to use when selecting a move for each step of
        simulation.  Larger values produce a broader probability distribution and
        hence more exploration.  Smaller values produce a stronger preference for
        whatever action did best in the tree search.
      puct_scale: float
        the scale of the PUCT term in the expression for selecting actions during
        tree search.  This should be roughly similar in magnitude to the rewards
        given by the environment, since the PUCT term is added to the mean
        discounted reward.  This may be None, in which case a value is adaptively
        selected that tries to match the mean absolute value of the discounted
        reward.
      max_checkpoints_to_keep: int
        the maximum number of checkpoint files to keep.  When this number is reached, older
        files are deleted.
      checkpoint_interval: float
        the time interval at which to save checkpoints, measured in seconds
      restore: bool
        if True, restore the model from the most recent checkpoint and continue training
        from there.  If False, retrain the model from scratch.
      """
      if puct_scale is None:
          self._puct_scale = 1.0
          adapt_puct = True
      else:
          self._puct_scale = puct_scale
          adapt_puct = False
      with self._graph._get_tf("Graph").as_default():
          self._graph.session.run(tf.global_variables_initializer())
          if restore:
              self.restore()
          variables = tf.get_collection(
              tf.GraphKeys.GLOBAL_VARIABLES, scope="global"
          )
          manager = tf.train.CheckpointManager(
              self._checkpoint,
              self._graph.model_dir,
              max_checkpoints_to_keep,
          )
          self._checkpoint_time = time.time() + checkpoint_interval
  
          # Run the algorithm.
  
          for iteration in range(iterations):
              buffer = self._run_episodes(
                  steps_per_iteration, temperature, manager, adapt_puct
              )
              self._optimize_policy(buffer, epochs_per_iteration)
  
          # Save a file checkpoint.
  
          with self._graph.session.as_default():
              manager.save()
  
imports: from deepchem.models import TensorGraph
  from deepchem.models.optimizers import Adam
  from deepchem.models.tensorgraph.layers import Feature, Weights, Label, Layer
  import numpy
  import tensorflow
  import copy
  import time
label_desc: Use image node_id 3 for calling the MCTS obj's underlying member method code with example usage: obj.fit(iterations, steps_per_iteration, epochs_per_iteration, temperature, puct_scale, max_checkpoints_to_keep, checkpoint_interval, restore) without return types