class_docstr: A DDPG model based on https://arxiv.org/pdf/1509.02971.pdf.
  
  DDPG, also known as Deep Deterministic Policy Gradient, uses actor-critic
  method to optimize the policy and reward prediction. It uses a supervised
  method to update the critic network and policy gradient to update the actor
  network. And there are exploration strategy, replay buffer and target
  networks involved to stabilize the training process.
  
  Example:
      $ python garage/examples/tf/ddpg_pendulum.py
  
  Args:
      env_spec (EnvSpec): Environment specification.
      policy (Policy): Policy.
      qf (object): The q value network.
      replay_buffer (ReplayBuffer): Replay buffer.
      sampler (garage.sampler.Sampler): Sampler.
      steps_per_epoch (int): Number of train_once calls per epoch.
      n_train_steps (int): Training steps.
      buffer_batch_size (int): Batch size of replay buffer.
      max_episode_length_eval (int or None): Maximum length of episodes used
          for off-policy evaluation. If None, defaults to
          `env_spec.max_episode_length`.
      min_buffer_size (int): The minimum buffer size for replay buffer.
      exploration_policy (ExplorationPolicy): Exploration strategy.
      target_update_tau (float): Interpolation parameter for doing the
          soft target update.
      policy_lr (float): Learning rate for training policy network.
      qf_lr (float): Learning rate for training q value network.
      discount(float): Discount factor for the cumulative return.
      policy_weight_decay (float): L2 regularization factor for parameters
          of the policy network. Value of 0 means no regularization.
      qf_weight_decay (float): L2 regularization factor for parameters
          of the q value network. Value of 0 means no regularization.
      policy_optimizer (tf.compat.v1.train.Optimizer): Optimizer for training
          policy network.
      qf_optimizer (tf.compat.v1.train.Optimizer): Optimizer for training
          Q-function network.
      clip_pos_returns (bool): Whether or not clip positive returns.
      clip_return (float): Clip return to be in [-clip_return,
          clip_return].
      max_action (float): Maximum action magnitude.
      reward_scale (float): Reward scale.
      name (str): Name of the algorithm shown in computation graph.
class_docstr_tok: ["A","DDPG","model","based","on","https",":","\/\/arxiv.org\/pdf\/1509.02971.pdf",".","DDPG",",","also","known","as","Deep","Deterministic","Policy","Gradient",",","uses","actor-critic","method","to","optimize","the","policy","and","reward","prediction",".","It","uses","a","supervised","method","to","update","the","critic","network","and","policy","gradient","to","update","the","actor","network",".","And","there","are","exploration","strategy",",","replay","buffer","and","target","networks","involved","to","stabilize","the","training","process",".","Example",":","$","python","garage\/examples\/tf\/ddpg_pendulum.py","Args",":","env_spec","(","EnvSpec",")",":","Environment","specification",".","policy","(","Policy",")",":","Policy",".","qf","(","object",")",":","The","q","value","network",".","replay_buffer","(","ReplayBuffer",")",":","Replay","buffer",".","sampler","(","garage.sampler.Sampler",")",":","Sampler",".","steps_per_epoch","(","int",")",":","Number","of","train_once","calls","per","epoch",".","n_train_steps","(","int",")",":","Training","steps",".","buffer_batch_size","(","int",")",":","Batch","size","of","replay","buffer",".","max_episode_length_eval","(","int","or","None",")",":","Maximum","length","of","episodes","used","for","off-policy","evaluation",".","If","None",",","defaults","to","`","env_spec.max_episode_length","`",".","min_buffer_size","(","int",")",":","The","minimum","buffer","size","for","replay","buffer",".","exploration_policy","(","ExplorationPolicy",")",":","Exploration","strategy",".","target_update_tau","(","float",")",":","Interpolation","parameter","for","doing","the","soft","target","update",".","policy_lr","(","float",")",":","Learning","rate","for","training","policy","network",".","qf_lr","(","float",")",":","Learning","rate","for","training","q","value","network",".","discount","(","float",")",":","Discount","factor","for","the","cumulative","return",".","policy_weight_decay","(","float",")",":","L2","regularization","factor","for","parameters","of","the","policy","network",".","Value","of","0","means","no","regularization",".","qf_weight_decay","(","float",")",":","L2","regularization","factor","for","parameters","of","the","q","value","network",".","Value","of","0","means","no","regularization",".","policy_optimizer","(","tf.compat.v1.train.Optimizer",")",":","Optimizer","for","training","policy","network",".","qf_optimizer","(","tf.compat.v1.train.Optimizer",")",":","Optimizer","for","training","Q-function","network",".","clip_pos_returns","(","bool",")",":","Whether","or","not","clip","positive","returns",".","clip_return","(","float",")",":","Clip","return","to","be","in","[","-clip_return",",","clip_return","]",".","max_action","(","float",")",":","Maximum","action","magnitude",".","reward_scale","(","float",")",":","Reward","scale",".","name","(","str",")",":","Name","of","the","algorithm","shown","in","computation","graph","."]
docstr: Perform one step of policy optimization given one batch of samples.
  
  Args:
      itr (int): Iteration number.
      episodes (EpisodeBatch): Batch of episodes.
docstr_tok: ["Perform","one","step","of","policy","optimization","given","one","batch","of","samples",".","Args",":","itr","(","int",")",":","Iteration","number",".","episodes","(","EpisodeBatch",")",":","Batch","of","episodes","."]
code: def _train_once(self, itr, episodes):
      """Perform one step of policy optimization given one batch of samples.
  
      Args:
          itr (int): Iteration number.
          episodes (EpisodeBatch): Batch of episodes.
  
      """
      self._replay_buffer.add_episode_batch(episodes)
      epoch = itr / self._steps_per_epoch
  
      for _ in range(self._n_train_steps):
          if (
              self._replay_buffer.n_transitions_stored
              >= self._min_buffer_size
          ):
              qf_loss, y_s, qval, policy_loss = self._optimize_policy()
  
              self._episode_policy_losses.append(policy_loss)
              self._episode_qf_losses.append(qf_loss)
              self._epoch_ys.append(y_s)
              self._epoch_qs.append(qval)
  
      if itr % self._steps_per_epoch == 0:
          logger.log("Training finished")
  
          if (
              self._replay_buffer.n_transitions_stored
              >= self._min_buffer_size
          ):
              tabular.record("Epoch", epoch)
              tabular.record(
                  "Policy/AveragePolicyLoss",
                  np.mean(self._episode_policy_losses),
              )
              tabular.record(
                  "QFunction/AverageQFunctionLoss",
                  np.mean(self._episode_qf_losses),
              )
              tabular.record(
                  "QFunction/AverageQ", np.mean(self._epoch_qs)
              )
              tabular.record("QFunction/MaxQ", np.max(self._epoch_qs))
              tabular.record(
                  "QFunction/AverageAbsQ",
                  np.mean(np.abs(self._epoch_qs)),
              )
              tabular.record(
                  "QFunction/AverageY", np.mean(self._epoch_ys)
              )
              tabular.record("QFunction/MaxY", np.max(self._epoch_ys))
              tabular.record(
                  "QFunction/AverageAbsY",
                  np.mean(np.abs(self._epoch_ys)),
              )
  
file_path: garage/src/garage/tf/algos/ddpg.py
imports: from dowel import logger, tabular
  import numpy
  import tensorflow
  from garage import _Default, log_performance, make_optimizer, obtain_evaluation_episodes
  from garage.np.algos import RLAlgorithm
  from garage.tf import compile_function, get_target_ops
label_desc: Use image node_id 6 for calling the DDPG obj's underlying member method code with example usage: obj._train_once(itr, episodes) without return types