class_docstr: DQN from https://arxiv.org/pdf/1312.5602.pdf.
  
  Known as Deep Q-Network, it estimates the Q-value function by deep neural
  networks. It enables Q-Learning to be applied on high complexity
  environments. To deal with pixel environments, numbers of tricks are
  usually needed, e.g. skipping frames and stacking frames as single
  observation.
  
  Args:
      env_spec (EnvSpec): Environment specification.
      policy (Policy): Policy.
      qf (object): The q value network.
      replay_buffer (ReplayBuffer): Replay buffer.
      sampler (garage.sampler.Sampler): Sampler.
      exploration_policy (ExplorationPolicy): Exploration strategy.
      steps_per_epoch (int): Number of train_once calls per epoch.
      min_buffer_size (int): The minimum buffer size for replay buffer.
      buffer_batch_size (int): Batch size for replay buffer.
      max_episode_length_eval (int or None): Maximum length of episodes used
          for off-policy evaluation. If None, defaults to
          `env_spec.max_episode_length`.
      n_train_steps (int): Training steps.
      qf_lr (float): Learning rate for Q-Function.
      qf_optimizer (tf.compat.v1.train.Optimizer): Optimizer for Q-Function.
      discount (float): Discount factor for rewards.
      target_network_update_freq (int): Frequency of updating target
          network.
      grad_norm_clipping (float): Maximum clipping value for clipping
          tensor values to a maximum L2-norm. It must be larger than 0.
          If None, no gradient clipping is done. For detail, see
          docstring for tf.clip_by_norm.
      double_q (bool): Bool for using double q-network.
      reward_scale (float): Reward scale.
      name (str): Name of the algorithm.
class_docstr_tok: ["DQN","from","https",":","\/\/arxiv.org\/pdf\/1312.5602.pdf",".","Known","as","Deep","Q-Network",",","it","estimates","the","Q-value","function","by","deep","neural","networks",".","It","enables","Q-Learning","to","be","applied","on","high","complexity","environments",".","To","deal","with","pixel","environments",",","numbers","of","tricks","are","usually","needed",",","e.g",".","skipping","frames","and","stacking","frames","as","single","observation",".","Args",":","env_spec","(","EnvSpec",")",":","Environment","specification",".","policy","(","Policy",")",":","Policy",".","qf","(","object",")",":","The","q","value","network",".","replay_buffer","(","ReplayBuffer",")",":","Replay","buffer",".","sampler","(","garage.sampler.Sampler",")",":","Sampler",".","exploration_policy","(","ExplorationPolicy",")",":","Exploration","strategy",".","steps_per_epoch","(","int",")",":","Number","of","train_once","calls","per","epoch",".","min_buffer_size","(","int",")",":","The","minimum","buffer","size","for","replay","buffer",".","buffer_batch_size","(","int",")",":","Batch","size","for","replay","buffer",".","max_episode_length_eval","(","int","or","None",")",":","Maximum","length","of","episodes","used","for","off-policy","evaluation",".","If","None",",","defaults","to","`","env_spec.max_episode_length","`",".","n_train_steps","(","int",")",":","Training","steps",".","qf_lr","(","float",")",":","Learning","rate","for","Q-Function",".","qf_optimizer","(","tf.compat.v1.train.Optimizer",")",":","Optimizer","for","Q-Function",".","discount","(","float",")",":","Discount","factor","for","rewards",".","target_network_update_freq","(","int",")",":","Frequency","of","updating","target","network",".","grad_norm_clipping","(","float",")",":","Maximum","clipping","value","for","clipping","tensor","values","to","a","maximum","L2-norm",".","It","must","be","larger","than","0",".","If","None",",","no","gradient","clipping","is","done",".","For","detail",",","see","docstring","for","tf.clip_by_norm",".","double_q","(","bool",")",":","Bool","for","using","double","q-network",".","reward_scale","(","float",")",":","Reward","scale",".","name","(","str",")",":","Name","of","the","algorithm","."]
docstr: Initialize the networks and Ops.
  
  Assume discrete space for dqn, so action dimension
  will always be action_space.n
docstr_tok: ["Initialize","the","networks","and","Ops",".","Assume","discrete","space","for","dqn",",","so","action","dimension","will","always","be","action_space.n"]
code: def _init_opt(self):
      """Initialize the networks and Ops.
  
      Assume discrete space for dqn, so action dimension
      will always be action_space.n
      """
      action_dim = self._env_spec.action_space.n
  
      # build q networks
      with tf.name_scope(self._name):
          action_t_ph = tf.compat.v1.placeholder(
              tf.int32, None, name="action"
          )
          reward_t_ph = tf.compat.v1.placeholder(
              tf.float32, None, name="reward"
          )
          done_t_ph = tf.compat.v1.placeholder(
              tf.float32, None, name="done"
          )
  
          with tf.name_scope("update_ops"):
              target_update_op = get_target_ops(
                  self._qf.get_global_vars(),
                  self._target_qf.get_global_vars(),
              )
  
          self._qf_update_ops = compile_function(
              inputs=[], outputs=target_update_op
          )
  
          with tf.name_scope("td_error"):
              # Q-value of the selected action
              action = tf.one_hot(
                  action_t_ph, action_dim, on_value=1.0, off_value=0.0
              )
              q_selected = tf.reduce_sum(
                  self._qf.q_vals * action, axis=1  # yapf: disable
              )
  
              # r + Q'(s', argmax_a(Q(s', _)) - Q(s, a)
              if self._double_q:
                  target_qval_with_online_q = self._qf.build(
                      self._target_qf.input, self._qf.name
                  )
                  future_best_q_val_action = tf.argmax(
                      target_qval_with_online_q, 1
                  )
                  future_best_q_val = tf.reduce_sum(
                      self._target_qf.q_vals
                      * tf.one_hot(
                          future_best_q_val_action,
                          action_dim,
                          on_value=1.0,
                          off_value=0.0,
                      ),
                      axis=1,
                  )
              else:
                  # r + max_a(Q'(s', _)) - Q(s, a)
                  future_best_q_val = tf.reduce_max(
                      self._target_qf.q_vals, axis=1
                  )
  
              q_best_masked = (1.0 - done_t_ph) * future_best_q_val
              # if done, it's just reward
              # else reward + discount * future_best_q_val
              target_q_values = (
                  reward_t_ph + self._discount * q_best_masked
              )
  
              # td_error = q_selected - tf.stop_gradient(target_q_values)
              loss = tf.compat.v1.losses.huber_loss(
                  q_selected, tf.stop_gradient(target_q_values)
              )
              loss = tf.reduce_mean(loss)
  
          with tf.name_scope("optimize_ops"):
              qf_optimizer = make_optimizer(
                  self._qf_optimizer, learning_rate=self._qf_lr
              )
              if self._grad_norm_clipping is not None:
                  gradients = qf_optimizer.compute_gradients(
                      loss, var_list=self._qf.get_trainable_vars()
                  )
                  for i, (grad, var) in enumerate(gradients):
                      if grad is not None:
                          gradients[i] = (
                              tf.clip_by_norm(
                                  grad, self._grad_norm_clipping
                              ),
                              var,
                          )
                      optimize_loss = qf_optimizer.apply_gradients(
                          gradients
                      )
              else:
                  optimize_loss = qf_optimizer.minimize(
                      loss, var_list=self._qf.get_trainable_vars()
                  )
  
          self._train_qf = compile_function(
              inputs=[
                  self._qf.input,
                  action_t_ph,
                  reward_t_ph,
                  done_t_ph,
                  self._target_qf.input,
              ],
              outputs=[loss, optimize_loss],
          )
  
file_path: garage/src/garage/tf/algos/dqn.py
imports: import akro
  from dowel import tabular
  import numpy
  import tensorflow
  from garage import log_performance, make_optimizer, obtain_evaluation_episodes
  from garage.np.algos import RLAlgorithm
  from garage.tf import compile_function, get_target_ops
label_desc: Use image node_id 2 for calling the DQN obj's underlying member method code with example usage: obj._init_opt() without return types