class_docstr: A DDPG Agent.
docstr: Creates a DDPG Agent.
  
  Args:
    time_step_spec: A `TimeStep` spec of the expected time_steps.
    action_spec: A nest of BoundedTensorSpec representing the actions.
    actor_network: A tf_agents.network.Network to be used by the agent. The
      network will be called with call(observation, step_type[, policy_state])
      and should return (action, new_state).
    critic_network: A tf_agents.network.Network to be used by the agent. The
      network will be called with call((observation, action), step_type[,
      policy_state]) and should return (q_value, new_state).
    actor_optimizer: The optimizer to use for the actor network.
    critic_optimizer: The optimizer to use for the critic network.
    ou_stddev: Standard deviation for the Ornstein-Uhlenbeck (OU) noise added
      in the default collect policy.
    ou_damping: Damping factor for the OU noise added in the default collect
      policy.
    target_actor_network: (Optional.)  A `tf_agents.network.Network` to be
      used as the actor target network during Q learning.  Every
      `target_update_period` train steps, the weights from `actor_network` are
      copied (possibly withsmoothing via `target_update_tau`) to `
      target_q_network`.  If `target_actor_network` is not provided, it is
      created by making a copy of `actor_network`, which initializes a new
      network with the same structure and its own layers and weights.
      Performing a `Network.copy` does not work when the network instance
      already has trainable parameters (e.g., has already been built, or when
      the network is sharing layers with another).  In these cases, it is up
      to you to build a copy having weights that are not shared with the
      original `actor_network`, so that this can be used as a target network.
      If you provide a `target_actor_network` that shares any weights with
      `actor_network`, a warning will be logged but no exception is thrown.
    target_critic_network: (Optional.) Similar network as target_actor_network
      but for the critic_network. See documentation for target_actor_network.
    target_update_tau: Factor for soft update of the target networks.
    target_update_period: Period for soft update of the target networks.
    dqda_clipping: when computing the actor loss, clips the gradient dqda
      element-wise between [-dqda_clipping, dqda_clipping]. Does not perform
      clipping if dqda_clipping == 0.
    td_errors_loss_fn:  A function for computing the TD errors loss. If None,
      a default value of elementwise huber_loss is used.
    gamma: A discount factor for future rewards.
    reward_scale_factor: Multiplicative scale for the reward.
    gradient_clipping: Norm length to clip gradients.
    debug_summaries: A bool to gather debug summaries.
    summarize_grads_and_vars: If True, gradient and network variable summaries
      will be written during training.
    train_step_counter: An optional counter to increment every time the train
      op is run.  Defaults to the global_step.
    name: The name of this agent. All variables in this module will fall under
      that name. Defaults to the class name.
docstr_tok: ["Creates","a","DDPG","Agent",".","Args",":","time_step_spec",":","A","`","TimeStep","`","spec","of","the","expected","time_steps",".","action_spec",":","A","nest","of","BoundedTensorSpec","representing","the","actions",".","actor_network",":","A","tf_agents.network.Network","to","be","used","by","the","agent",".","The","network","will","be","called","with","call","(","observation",",","step_type","[",",","policy_state","]",")","and","should","return","(","action",",","new_state",")",".","critic_network",":","A","tf_agents.network.Network","to","be","used","by","the","agent",".","The","network","will","be","called","with","call","(","(","observation",",","action",")",",","step_type","[",",","policy_state","]",")","and","should","return","(","q_value",",","new_state",")",".","actor_optimizer",":","The","optimizer","to","use","for","the","actor","network",".","critic_optimizer",":","The","optimizer","to","use","for","the","critic","network",".","ou_stddev",":","Standard","deviation","for","the","Ornstein-Uhlenbeck","(","OU",")","noise","added","in","the","default","collect","policy",".","ou_damping",":","Damping","factor","for","the","OU","noise","added","in","the","default","collect","policy",".","target_actor_network",":","(","Optional",".",")","A","`","tf_agents.network.Network","`","to","be","used","as","the","actor","target","network","during","Q","learning",".","Every","`","target_update_period","`","train","steps",",","the","weights","from","`","actor_network","`","are","copied","(","possibly","withsmoothing","via","`","target_update_tau","`",")","to","`","target_q_network","`",".","If","`","target_actor_network","`","is","not","provided",",","it","is","created","by","making","a","copy","of","`","actor_network","`",",","which","initializes","a","new","network","with","the","same","structure","and","its","own","layers","and","weights",".","Performing","a","`","Network.copy","`","does","not","work","when","the","network","instance","already","has","trainable","parameters","(","e.g.",",","has","already","been","built",",","or","when","the","network","is","sharing","layers","with","another",")",".","In","these","cases",",","it","is","up","to","you","to","build","a","copy","having","weights","that","are","not","shared","with","the","original","`","actor_network","`",",","so","that","this","can","be","used","as","a","target","network",".","If","you","provide","a","`","target_actor_network","`","that","shares","any","weights","with","`","actor_network","`",",","a","warning","will","be","logged","but","no","exception","is","thrown",".","target_critic_network",":","(","Optional",".",")","Similar","network","as","target_actor_network","but","for","the","critic_network",".","See","documentation","for","target_actor_network",".","target_update_tau",":","Factor","for","soft","update","of","the","target","networks",".","target_update_period",":","Period","for","soft","update","of","the","target","networks",".","dqda_clipping",":","when","computing","the","actor","loss",",","clips","the","gradient","dqda","element-wise","between","[","-dqda_clipping",",","dqda_clipping","]",".","Does","not","perform","clipping","if","dqda_clipping","==","0.","td_errors_loss_fn",":","A","function","for","computing","the","TD","errors","loss",".","If","None",",","a","default","value","of","elementwise","huber_loss","is","used",".","gamma",":","A","discount","factor","for","future","rewards",".","reward_scale_factor",":","Multiplicative","scale","for","the","reward",".","gradient_clipping",":","Norm","length","to","clip","gradients",".","debug_summaries",":","A","bool","to","gather","debug","summaries",".","summarize_grads_and_vars",":","If","True",",","gradient","and","network","variable","summaries","will","be","written","during","training",".","train_step_counter",":","An","optional","counter","to","increment","every","time","the","train","op","is","run",".","Defaults","to","the","global_step",".","name",":","The","name","of","this","agent",".","All","variables","in","this","module","will","fall","under","that","name",".","Defaults","to","the","class","name","."]
code: def __init__(
      self,
      time_step_spec: ts.TimeStep,
      action_spec: types.NestedTensorSpec,
      actor_network: network.Network,
      critic_network: network.Network,
      actor_optimizer: Optional[types.Optimizer] = None,
      critic_optimizer: Optional[types.Optimizer] = None,
      ou_stddev: types.Float = 1.0,
      ou_damping: types.Float = 1.0,
      target_actor_network: Optional[network.Network] = None,
      target_critic_network: Optional[network.Network] = None,
      target_update_tau: types.Float = 1.0,
      target_update_period: types.Int = 1,
      dqda_clipping: Optional[types.Float] = None,
      td_errors_loss_fn: Optional[types.LossFn] = None,
      gamma: types.Float = 1.0,
      reward_scale_factor: types.Float = 1.0,
      gradient_clipping: Optional[types.Float] = None,
      debug_summaries: bool = False,
      summarize_grads_and_vars: bool = False,
      train_step_counter: Optional[tf.Variable] = None,
      name: Optional[Text] = None,
  ):
      """Creates a DDPG Agent.
  
      Args:
        time_step_spec: A `TimeStep` spec of the expected time_steps.
        action_spec: A nest of BoundedTensorSpec representing the actions.
        actor_network: A tf_agents.network.Network to be used by the agent. The
          network will be called with call(observation, step_type[, policy_state])
          and should return (action, new_state).
        critic_network: A tf_agents.network.Network to be used by the agent. The
          network will be called with call((observation, action), step_type[,
          policy_state]) and should return (q_value, new_state).
        actor_optimizer: The optimizer to use for the actor network.
        critic_optimizer: The optimizer to use for the critic network.
        ou_stddev: Standard deviation for the Ornstein-Uhlenbeck (OU) noise added
          in the default collect policy.
        ou_damping: Damping factor for the OU noise added in the default collect
          policy.
        target_actor_network: (Optional.)  A `tf_agents.network.Network` to be
          used as the actor target network during Q learning.  Every
          `target_update_period` train steps, the weights from `actor_network` are
          copied (possibly withsmoothing via `target_update_tau`) to `
          target_q_network`.  If `target_actor_network` is not provided, it is
          created by making a copy of `actor_network`, which initializes a new
          network with the same structure and its own layers and weights.
          Performing a `Network.copy` does not work when the network instance
          already has trainable parameters (e.g., has already been built, or when
          the network is sharing layers with another).  In these cases, it is up
          to you to build a copy having weights that are not shared with the
          original `actor_network`, so that this can be used as a target network.
          If you provide a `target_actor_network` that shares any weights with
          `actor_network`, a warning will be logged but no exception is thrown.
        target_critic_network: (Optional.) Similar network as target_actor_network
          but for the critic_network. See documentation for target_actor_network.
        target_update_tau: Factor for soft update of the target networks.
        target_update_period: Period for soft update of the target networks.
        dqda_clipping: when computing the actor loss, clips the gradient dqda
          element-wise between [-dqda_clipping, dqda_clipping]. Does not perform
          clipping if dqda_clipping == 0.
        td_errors_loss_fn:  A function for computing the TD errors loss. If None,
          a default value of elementwise huber_loss is used.
        gamma: A discount factor for future rewards.
        reward_scale_factor: Multiplicative scale for the reward.
        gradient_clipping: Norm length to clip gradients.
        debug_summaries: A bool to gather debug summaries.
        summarize_grads_and_vars: If True, gradient and network variable summaries
          will be written during training.
        train_step_counter: An optional counter to increment every time the train
          op is run.  Defaults to the global_step.
        name: The name of this agent. All variables in this module will fall under
          that name. Defaults to the class name.
      """
      tf.Module.__init__(self, name=name)
      self._actor_network = actor_network
      actor_network.create_variables(time_step_spec.observation)
      if target_actor_network:
          target_actor_network.create_variables(
              time_step_spec.observation
          )
      self._target_actor_network = (
          common.maybe_copy_target_network_with_checks(
              self._actor_network,
              target_actor_network,
              "TargetActorNetwork",
              input_spec=time_step_spec.observation,
          )
      )
  
      self._critic_network = critic_network
      critic_input_spec = (time_step_spec.observation, action_spec)
      critic_network.create_variables(critic_input_spec)
      if target_critic_network:
          target_critic_network.create_variables(critic_input_spec)
      self._target_critic_network = (
          common.maybe_copy_target_network_with_checks(
              self._critic_network,
              target_critic_network,
              "TargetCriticNetwork",
              input_spec=critic_input_spec,
          )
      )
  
      self._actor_optimizer = actor_optimizer
      self._critic_optimizer = critic_optimizer
  
      self._ou_stddev = ou_stddev
      self._ou_damping = ou_damping
      self._target_update_tau = target_update_tau
      self._target_update_period = target_update_period
      self._dqda_clipping = dqda_clipping
      self._td_errors_loss_fn = (
          td_errors_loss_fn or common.element_wise_huber_loss
      )
      self._gamma = gamma
      self._reward_scale_factor = reward_scale_factor
      self._gradient_clipping = gradient_clipping
  
      self._update_target = self._get_target_updater(
          target_update_tau, target_update_period
      )
  
      policy = actor_policy.ActorPolicy(
          time_step_spec=time_step_spec,
          action_spec=action_spec,
          actor_network=self._actor_network,
          clip=True,
      )
      collect_policy = actor_policy.ActorPolicy(
          time_step_spec=time_step_spec,
          action_spec=action_spec,
          actor_network=self._actor_network,
          clip=False,
      )
      collect_policy = ou_noise_policy.OUNoisePolicy(
          collect_policy,
          ou_stddev=self._ou_stddev,
          ou_damping=self._ou_damping,
          clip=True,
      )
  
      super(DdpgAgent, self).__init__(
          time_step_spec,
          action_spec,
          policy,
          collect_policy,
          train_sequence_length=2
          if not self._actor_network.state_spec
          else None,
          debug_summaries=debug_summaries,
          summarize_grads_and_vars=summarize_grads_and_vars,
          train_step_counter=train_step_counter,
      )
  
      self._as_transition = data_converter.AsTransition(
          self.data_context,
          squeeze_time_dim=not self._actor_network.state_spec,
      )
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  import collections
  from typing import Optional, Text
  import gin
  import tensorflow
  from tf_agents.agents import data_converter
  from tf_agents.agents import tf_agent
  from tf_agents.networks import network
  from tf_agents.policies import actor_policy
  from tf_agents.policies import ou_noise_policy
  from tf_agents.trajectories import time_step
  from tf_agents.typing import types
  from tf_agents.utils import common
  from tf_agents.utils import eager_utils
  from tf_agents.utils import nest_utils
label_desc: Use image node_id 1 to create a new DdpgAgent object from inherited base classes: tf_agent with example: obj = DdpgAgent(time_step_spec, action_spec, actor_network, critic_network, actor_optimizer, critic_optimizer, ou_stddev, ou_damping, target_actor_network, target_critic_network, target_update_tau, target_update_period, dqda_clipping, td_errors_loss_fn, gamma, reward_scale_factor, gradient_clipping, debug_summaries, summarize_grads_and_vars, train_step_counter, name)