class_docstr: A DDPG model implemented with PyTorch.
  
  DDPG, also known as Deep Deterministic Policy Gradient, uses actor-critic
  method to optimize the policy and Q-function prediction. It uses a
  supervised method to update the critic network and policy gradient to
  update the actor network. And there are exploration strategy, replay
  buffer and target networks involved to stabilize the training process.
  
  Args:
      env_spec (EnvSpec): Environment specification.
      policy (garage.torch.policies.Policy): Policy.
      qf (object): Q-value network.
      replay_buffer (ReplayBuffer): Replay buffer.
      sampler (garage.sampler.Sampler): Sampler.
      steps_per_epoch (int): Number of train_once calls per epoch.
      n_train_steps (int): Training steps.
      buffer_batch_size (int): Batch size of replay buffer.
      min_buffer_size (int): The minimum buffer size for replay buffer.
      exploration_policy (garage.np.exploration_policies.ExplorationPolicy): # noqa: E501
              Exploration strategy.
      target_update_tau (float): Interpolation parameter for doing the
          soft target update.
      max_episode_length_eval (int or None): Maximum length of episodes used
          for off-policy evaluation. If `None`, defaults to
          `env_spec.max_episode_length`.
      discount(float): Discount factor for the cumulative return.
      policy_weight_decay (float): L2 weight decay factor for parameters
          of the policy network.
      qf_weight_decay (float): L2 weight decay factor for parameters
          of the q value network.
      policy_optimizer (Union[type, tuple[type, dict]]): Type of optimizer
          for training policy network. This can be an optimizer type such as
          `torch.optim.Adam` or a tuple of type and dictionary, where
          dictionary contains arguments to initialize the optimizer
          e.g. `(torch.optim.Adam, {'lr' : 1e-3})`.
      qf_optimizer (Union[type, tuple[type, dict]]): Type of optimizer
          for training Q-value network. This can be an optimizer type such
          as `torch.optim.Adam` or a tuple of type and dictionary, where
          dictionary contains arguments to initialize the optimizer
          e.g. `(torch.optim.Adam, {'lr' : 1e-3})`.
      policy_lr (float): Learning rate for policy network parameters.
      qf_lr (float): Learning rate for Q-value network parameters.
      clip_pos_returns (bool): Whether or not clip positive returns.
      clip_return (float): Clip return to be in [-clip_return,
          clip_return].
      max_action (float): Maximum action magnitude.
      reward_scale (float): Reward scale.
class_docstr_tok: ["A","DDPG","model","implemented","with","PyTorch",".","DDPG",",","also","known","as","Deep","Deterministic","Policy","Gradient",",","uses","actor-critic","method","to","optimize","the","policy","and","Q-function","prediction",".","It","uses","a","supervised","method","to","update","the","critic","network","and","policy","gradient","to","update","the","actor","network",".","And","there","are","exploration","strategy",",","replay","buffer","and","target","networks","involved","to","stabilize","the","training","process",".","Args",":","env_spec","(","EnvSpec",")",":","Environment","specification",".","policy","(","garage.torch.policies.Policy",")",":","Policy",".","qf","(","object",")",":","Q-value","network",".","replay_buffer","(","ReplayBuffer",")",":","Replay","buffer",".","sampler","(","garage.sampler.Sampler",")",":","Sampler",".","steps_per_epoch","(","int",")",":","Number","of","train_once","calls","per","epoch",".","n_train_steps","(","int",")",":","Training","steps",".","buffer_batch_size","(","int",")",":","Batch","size","of","replay","buffer",".","min_buffer_size","(","int",")",":","The","minimum","buffer","size","for","replay","buffer",".","exploration_policy","(","garage.np.exploration_policies.ExplorationPolicy",")",":","#","noqa",":","E501","Exploration","strategy",".","target_update_tau","(","float",")",":","Interpolation","parameter","for","doing","the","soft","target","update",".","max_episode_length_eval","(","int","or","None",")",":","Maximum","length","of","episodes","used","for","off-policy","evaluation",".","If","`","None","`",",","defaults","to","`","env_spec.max_episode_length","`",".","discount","(","float",")",":","Discount","factor","for","the","cumulative","return",".","policy_weight_decay","(","float",")",":","L2","weight","decay","factor","for","parameters","of","the","policy","network",".","qf_weight_decay","(","float",")",":","L2","weight","decay","factor","for","parameters","of","the","q","value","network",".","policy_optimizer","(","Union","[","type",",","tuple","[","type",",","dict","]","]",")",":","Type","of","optimizer","for","training","policy","network",".","This","can","be","an","optimizer","type","such","as","`","torch.optim.Adam","`","or","a","tuple","of","type","and","dictionary",",","where","dictionary","contains","arguments","to","initialize","the","optimizer","e.g",".","`","(","torch.optim.Adam",",","{","'lr","'",":","1e-3","}",")","`",".","qf_optimizer","(","Union","[","type",",","tuple","[","type",",","dict","]","]",")",":","Type","of","optimizer","for","training","Q-value","network",".","This","can","be","an","optimizer","type","such","as","`","torch.optim.Adam","`","or","a","tuple","of","type","and","dictionary",",","where","dictionary","contains","arguments","to","initialize","the","optimizer","e.g",".","`","(","torch.optim.Adam",",","{","'lr","'",":","1e-3","}",")","`",".","policy_lr","(","float",")",":","Learning","rate","for","policy","network","parameters",".","qf_lr","(","float",")",":","Learning","rate","for","Q-value","network","parameters",".","clip_pos_returns","(","bool",")",":","Whether","or","not","clip","positive","returns",".","clip_return","(","float",")",":","Clip","return","to","be","in","[","-clip_return",",","clip_return","]",".","max_action","(","float",")",":","Maximum","action","magnitude",".","reward_scale","(","float",")",":","Reward","scale","."]
code: def __init__(
      self,
      env_spec,
      policy,
      qf,
      replay_buffer,
      sampler,
      *,  # Everything after this is numbers.
      steps_per_epoch=20,
      n_train_steps=50,
      max_episode_length_eval=None,
      buffer_batch_size=64,
      min_buffer_size=int(1e4),
      exploration_policy=None,
      target_update_tau=0.01,
      discount=0.99,
      policy_weight_decay=0,
      qf_weight_decay=0,
      policy_optimizer=torch.optim.Adam,
      qf_optimizer=torch.optim.Adam,
      policy_lr=_Default(1e-4),
      qf_lr=_Default(1e-3),
      clip_pos_returns=False,
      clip_return=np.inf,
      max_action=None,
      reward_scale=1.0,
  ):
      action_bound = env_spec.action_space.high
      self._tau = target_update_tau
      self._policy_weight_decay = policy_weight_decay
      self._qf_weight_decay = qf_weight_decay
      self._clip_pos_returns = clip_pos_returns
      self._clip_return = clip_return
      self._max_action = (
          action_bound if max_action is None else max_action
      )
  
      self._steps_per_epoch = steps_per_epoch
      self._episode_policy_losses = []
      self._episode_qf_losses = []
      self._epoch_ys = []
      self._epoch_qs = []
  
      self._policy = policy
      self._qf = qf
      self._n_train_steps = n_train_steps
  
      self._min_buffer_size = min_buffer_size
      self._qf = qf
      self._steps_per_epoch = steps_per_epoch
      self._n_train_steps = n_train_steps
      self._buffer_batch_size = buffer_batch_size
      self._discount = discount
      self._reward_scale = reward_scale
      self.max_episode_length = env_spec.max_episode_length
      self._max_episode_length_eval = env_spec.max_episode_length
  
      if max_episode_length_eval is not None:
          self._max_episode_length_eval = max_episode_length_eval
  
      self.env_spec = env_spec
      self.replay_buffer = replay_buffer
      self.policy = policy
      self.exploration_policy = exploration_policy
  
      self._target_policy = copy.deepcopy(self.policy)
      self._target_qf = copy.deepcopy(self._qf)
      self._policy_optimizer = make_optimizer(
          policy_optimizer, module=self.policy, lr=policy_lr
      )
      self._qf_optimizer = make_optimizer(
          qf_optimizer, module=self._qf, lr=qf_lr
      )
      self._eval_env = None
      self._sampler = sampler
  
file_path: garage/src/garage/torch/algos/ddpg.py
imports: import copy
  from dowel import logger, tabular
  import numpy
  import torch
  from garage import _Default, log_performance, make_optimizer, obtain_evaluation_episodes
  from garage.np.algos import RLAlgorithm
  from garage.torch import as_torch_dict, torch_to_np
  from garage.torch._functions import zero_optim_grads
label_desc: Use image node_id 1 to create a new DDPG object from inherited base classes: RLAlgorithm with example: obj = DDPG(env_spec, policy, qf, replay_buffer, sampler)