class_docstr: An implementation of the DQN agent.
class_docstr_tok: ["An","implementation","of","the","DQN","agent","."]
docstr: Initializes the agent and constructs the components of its graph.
  
  Args:
    sess: `tf.compat.v1.Session`, for executing ops.
    num_actions: int, number of actions the agent can take at any state.
    observation_shape: tuple of ints describing the observation shape.
    observation_dtype: tf.DType, specifies the type of the observations. Note
      that if your inputs are continuous, you should set this to tf.float32.
    stack_size: int, number of frames to use in state stack.
    network: tf.Keras.Model, expecting 2 parameters: num_actions,
      network_type. A call to this object will return an instantiation of the
      network provided. The network returned can be run with different inputs
      to create different outputs. See
      dopamine.discrete_domains.atari_lib.NatureDQNNetwork as an example.
    gamma: float, discount factor with the usual RL meaning.
    update_horizon: int, horizon at which updates are performed, the 'n' in
      n-step update.
    min_replay_history: int, number of transitions that should be experienced
      before the agent begins training its value function.
    update_period: int, period between DQN updates.
    target_update_period: int, update period for the target network.
    epsilon_fn: function expecting 4 parameters:
      (decay_period, step, warmup_steps, epsilon). This function should return
      the epsilon value used for exploration during training.
    epsilon_train: float, the value to which the agent's epsilon is eventually
      decayed during training.
    epsilon_eval: float, epsilon used when evaluating the agent.
    epsilon_decay_period: int, length of the epsilon decay schedule.
    tf_device: str, Tensorflow device on which the agent's graph is executed.
    eval_mode: bool, True for evaluation and False for training.
    use_staging: bool, when True use a staging area to prefetch the next
      training batch, speeding training up by about 30%.
    max_tf_checkpoints_to_keep: int, the number of TensorFlow checkpoints to
      keep.
    optimizer: `tf.compat.v1.train.Optimizer`, for training the value
      function.
    summary_writer: SummaryWriter object for outputting training statistics.
      Summary writing disabled if set to None.
      May also be a str specifying the base directory, in which case the
      SummaryWriter will be created by the agent.
    summary_writing_frequency: int, frequency with which summaries will be
      written. Lower values will result in slower training.
    allow_partial_reload: bool, whether we allow reloading a partial agent
      (for instance, only the network parameters).
docstr_tok: ["Initializes","the","agent","and","constructs","the","components","of","its","graph",".","Args",":","sess",":","`","tf.compat.v1.Session","`",",","for","executing","ops",".","num_actions",":","int",",","number","of","actions","the","agent","can","take","at","any","state",".","observation_shape",":","tuple","of","ints","describing","the","observation","shape",".","observation_dtype",":","tf.DType",",","specifies","the","type","of","the","observations",".","Note","that","if","your","inputs","are","continuous",",","you","should","set","this","to","tf.float32",".","stack_size",":","int",",","number","of","frames","to","use","in","state","stack",".","network",":","tf.Keras.Model",",","expecting","2","parameters",":","num_actions",",","network_type",".","A","call","to","this","object","will","return","an","instantiation","of","the","network","provided",".","The","network","returned","can","be","run","with","different","inputs","to","create","different","outputs",".","See","dopamine.discrete_domains.atari_lib.NatureDQNNetwork","as","an","example",".","gamma",":","float",",","discount","factor","with","the","usual","RL","meaning",".","update_horizon",":","int",",","horizon","at","which","updates","are","performed",",","the","'n","'","in","n-step","update",".","min_replay_history",":","int",",","number","of","transitions","that","should","be","experienced","before","the","agent","begins","training","its","value","function",".","update_period",":","int",",","period","between","DQN","updates",".","target_update_period",":","int",",","update","period","for","the","target","network",".","epsilon_fn",":","function","expecting","4","parameters",":","(","decay_period",",","step",",","warmup_steps",",","epsilon",")",".","This","function","should","return","the","epsilon","value","used","for","exploration","during","training",".","epsilon_train",":","float",",","the","value","to","which","the","agent","'s","epsilon","is","eventually","decayed","during","training",".","epsilon_eval",":","float",",","epsilon","used","when","evaluating","the","agent",".","epsilon_decay_period",":","int",",","length","of","the","epsilon","decay","schedule",".","tf_device",":","str",",","Tensorflow","device","on","which","the","agent","'s","graph","is","executed",".","eval_mode",":","bool",",","True","for","evaluation","and","False","for","training",".","use_staging",":","bool",",","when","True","use","a","staging","area","to","prefetch","the","next","training","batch",",","speeding","training","up","by","about","30","%",".","max_tf_checkpoints_to_keep",":","int",",","the","number","of","TensorFlow","checkpoints","to","keep",".","optimizer",":","`","tf.compat.v1.train.Optimizer","`",",","for","training","the","value","function",".","summary_writer",":","SummaryWriter","object","for","outputting","training","statistics",".","Summary","writing","disabled","if","set","to","None",".","May","also","be","a","str","specifying","the","base","directory",",","in","which","case","the","SummaryWriter","will","be","created","by","the","agent",".","summary_writing_frequency",":","int",",","frequency","with","which","summaries","will","be","written",".","Lower","values","will","result","in","slower","training",".","allow_partial_reload",":","bool",",","whether","we","allow","reloading","a","partial","agent","(","for","instance",",","only","the","network","parameters",")","."]
code: def __init__(
      self,
      sess,
      num_actions,
      observation_shape=atari_lib.NATURE_DQN_OBSERVATION_SHAPE,
      observation_dtype=atari_lib.NATURE_DQN_DTYPE,
      stack_size=atari_lib.NATURE_DQN_STACK_SIZE,
      network=atari_lib.NatureDQNNetwork,
      gamma=0.99,
      update_horizon=1,
      min_replay_history=20000,
      update_period=4,
      target_update_period=8000,
      epsilon_fn=linearly_decaying_epsilon,
      epsilon_train=0.01,
      epsilon_eval=0.001,
      epsilon_decay_period=250000,
      tf_device="/cpu:*",
      eval_mode=False,
      use_staging=False,
      max_tf_checkpoints_to_keep=4,
      optimizer=tf.compat.v1.train.RMSPropOptimizer(
          learning_rate=0.00025,
          decay=0.95,
          momentum=0.0,
          epsilon=0.00001,
          centered=True,
      ),
      summary_writer=None,
      summary_writing_frequency=500,
      allow_partial_reload=False,
  ):
      """Initializes the agent and constructs the components of its graph.
  
      Args:
        sess: `tf.compat.v1.Session`, for executing ops.
        num_actions: int, number of actions the agent can take at any state.
        observation_shape: tuple of ints describing the observation shape.
        observation_dtype: tf.DType, specifies the type of the observations. Note
          that if your inputs are continuous, you should set this to tf.float32.
        stack_size: int, number of frames to use in state stack.
        network: tf.Keras.Model, expecting 2 parameters: num_actions,
          network_type. A call to this object will return an instantiation of the
          network provided. The network returned can be run with different inputs
          to create different outputs. See
          dopamine.discrete_domains.atari_lib.NatureDQNNetwork as an example.
        gamma: float, discount factor with the usual RL meaning.
        update_horizon: int, horizon at which updates are performed, the 'n' in
          n-step update.
        min_replay_history: int, number of transitions that should be experienced
          before the agent begins training its value function.
        update_period: int, period between DQN updates.
        target_update_period: int, update period for the target network.
        epsilon_fn: function expecting 4 parameters:
          (decay_period, step, warmup_steps, epsilon). This function should return
          the epsilon value used for exploration during training.
        epsilon_train: float, the value to which the agent's epsilon is eventually
          decayed during training.
        epsilon_eval: float, epsilon used when evaluating the agent.
        epsilon_decay_period: int, length of the epsilon decay schedule.
        tf_device: str, Tensorflow device on which the agent's graph is executed.
        eval_mode: bool, True for evaluation and False for training.
        use_staging: bool, when True use a staging area to prefetch the next
          training batch, speeding training up by about 30%.
        max_tf_checkpoints_to_keep: int, the number of TensorFlow checkpoints to
          keep.
        optimizer: `tf.compat.v1.train.Optimizer`, for training the value
          function.
        summary_writer: SummaryWriter object for outputting training statistics.
          Summary writing disabled if set to None.
          May also be a str specifying the base directory, in which case the
          SummaryWriter will be created by the agent.
        summary_writing_frequency: int, frequency with which summaries will be
          written. Lower values will result in slower training.
        allow_partial_reload: bool, whether we allow reloading a partial agent
          (for instance, only the network parameters).
      """
      assert isinstance(observation_shape, tuple)
      logging.info(
          "Creating %s agent with the following parameters:",
          self.__class__.__name__,
      )
      logging.info("\t gamma: %f", gamma)
      logging.info("\t update_horizon: %f", update_horizon)
      logging.info("\t min_replay_history: %d", min_replay_history)
      logging.info("\t update_period: %d", update_period)
      logging.info("\t target_update_period: %d", target_update_period)
      logging.info("\t epsilon_train: %f", epsilon_train)
      logging.info("\t epsilon_eval: %f", epsilon_eval)
      logging.info("\t epsilon_decay_period: %d", epsilon_decay_period)
      logging.info("\t tf_device: %s", tf_device)
      logging.info("\t use_staging: %s", use_staging)
      logging.info("\t optimizer: %s", optimizer)
      logging.info(
          "\t max_tf_checkpoints_to_keep: %d",
          max_tf_checkpoints_to_keep,
      )
  
      self.num_actions = num_actions
      self.observation_shape = tuple(observation_shape)
      self.observation_dtype = observation_dtype
      self.stack_size = stack_size
      self.network = network
      self.gamma = gamma
      self.update_horizon = update_horizon
      self.cumulative_gamma = math.pow(gamma, update_horizon)
      self.min_replay_history = min_replay_history
      self.target_update_period = target_update_period
      self.epsilon_fn = epsilon_fn
      self.epsilon_train = epsilon_train
      self.epsilon_eval = epsilon_eval
      self.epsilon_decay_period = epsilon_decay_period
      self.update_period = update_period
      self.eval_mode = eval_mode
      self.training_steps = 0
      self.optimizer = optimizer
      tf.compat.v1.disable_v2_behavior()
      if isinstance(
          summary_writer, str
      ):  # If we're passing in directory name.
          self.summary_writer = tf.compat.v1.summary.FileWriter(
              summary_writer
          )
      else:
          self.summary_writer = summary_writer
      self.summary_writing_frequency = summary_writing_frequency
      self.allow_partial_reload = allow_partial_reload
  
      if sess is None:
          config = tf.compat.v1.ConfigProto(allow_soft_placement=True)
          # Allocate only subset of the GPU memory as needed which allows for
          # running multiple agents/workers on the same GPU.
          config.gpu_options.allow_growth = True
          self._sess = tf.compat.v1.Session("", config=config)
      else:
          self._sess = sess
  
      with tf.device(tf_device):
          # Create a placeholder for the state input to the DQN network.
          # The last axis indicates the number of consecutive frames stacked.
          state_shape = (1,) + self.observation_shape + (stack_size,)
          self.state = np.zeros(state_shape)
          self.state_ph = tf.compat.v1.placeholder(
              self.observation_dtype, state_shape, name="state_ph"
          )
          self._replay = self._build_replay_buffer(use_staging)
  
          self._build_networks()
  
          self._train_op = self._build_train_op()
          self._sync_qt_ops = self._build_sync_op()
  
      if self.summary_writer is not None:
          # All tf.summaries should have been defined prior to running this.
          self._merged_summaries = tf.compat.v1.summary.merge_all()
  
      var_map = atari_lib.maybe_transform_variable_names(
          tf.compat.v1.global_variables()
      )
      self._saver = tf.compat.v1.train.Saver(
          var_list=var_map, max_to_keep=max_tf_checkpoints_to_keep
      )
  
      # Variables to be initialized by the agent once it interacts with the
      # environment.
      self._observation = None
      self._last_observation = None
  
      if self.summary_writer is not None:
          self.summary_writer.add_graph(
              graph=tf.compat.v1.get_default_graph()
          )
      self._sess.run(tf.compat.v1.global_variables_initializer())
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  import math
  import os
  import random
  from absl import logging
  from dopamine.discrete_domains import atari_lib
  from dopamine.replay_memory import circular_replay_buffer
  import gin.tf
  import numpy
  import tensorflow
label_desc: Use image node_id 1 to create a new DQNAgent object from inherited base classes: object with example: obj = DQNAgent(sess, num_actions, observation_shape, observation_dtype, stack_size, network, gamma, update_horizon, min_replay_history, update_period, target_update_period, epsilon_fn, epsilon_train, epsilon_eval, epsilon_decay_period, tf_device, eval_mode, use_staging, max_tf_checkpoints_to_keep, optimizer, summary_writer, summary_writing_frequency, allow_partial_reload)