class_docstr: An ActorPolicy that also returns policy_info needed for PPO training.
  
  This policy requires two networks: the usual `actor_network` and the
  additional `value_network`. The value network can be executed with the
  `apply_value_network()` method.
  
  When the networks have state (RNNs, LSTMs) you must be careful to pass the
  state for the actor network to `action()` and the state of the value network
  to `apply_value_network()`. Use `get_initial_value_state()` to access
  the state of the value network.
docstr: Builds a PPO Policy given network Templates or functions.
  
  Args:
    time_step_spec: A `TimeStep` spec of the expected time_steps.
    action_spec: A nest of BoundedTensorSpec representing the actions.
    actor_network: An instance of a tf_agents.networks.network.Network, with
      call(observation, step_type, network_state).  Network should return one
      of the following: 1. a nested tuple of tfp.distributions objects
      matching action_spec, or 2. a nested tuple of tf.Tensors representing
      actions.
    value_network:  An instance of a tf_agents.networks.network.Network, with
      call(observation, step_type, network_state).  Network should return
      value predictions for the input state.
    observation_normalizer: An object to use for obervation normalization.
    clip: Whether to clip actions to spec before returning them.  Default
      True. Most policy-based algorithms (PCL, PPO, REINFORCE) use unclipped
      continuous actions for training.
    collect: If True, creates ops for actions_log_prob, value_preds, and
      action_distribution_params. (default True)
    compute_value_and_advantage_in_train: A bool to indicate where value
      prediction and advantage calculation happen.  If True, both happen in
      agent.train(), therefore no need to save the value prediction inside of
      policy info. If False, value prediction is computed during data
      collection. This argument must be set to `False` if mini batch learning
      is enabled.
  
  Raises:
    TypeError: if `actor_network` or `value_network` is not of type
      `tf_agents.networks.Network`.
    ValueError: if `actor_network` or `value_network` do not emit
      valid outputs.  For example, `actor_network` must either be
      a (legacy style) `DistributionNetwork`, or explicitly emit
      a nest of `tfp.distribution.Distribution` objects.
docstr_tok: ["Builds","a","PPO","Policy","given","network","Templates","or","functions",".","Args",":","time_step_spec",":","A","`","TimeStep","`","spec","of","the","expected","time_steps",".","action_spec",":","A","nest","of","BoundedTensorSpec","representing","the","actions",".","actor_network",":","An","instance","of","a","tf_agents.networks.network.Network",",","with","call","(","observation",",","step_type",",","network_state",")",".","Network","should","return","one","of","the","following",":","1.","a","nested","tuple","of","tfp.distributions","objects","matching","action_spec",",","or","2.","a","nested","tuple","of","tf.Tensors","representing","actions",".","value_network",":","An","instance","of","a","tf_agents.networks.network.Network",",","with","call","(","observation",",","step_type",",","network_state",")",".","Network","should","return","value","predictions","for","the","input","state",".","observation_normalizer",":","An","object","to","use","for","obervation","normalization",".","clip",":","Whether","to","clip","actions","to","spec","before","returning","them",".","Default","True",".","Most","policy-based","algorithms","(","PCL",",","PPO",",","REINFORCE",")","use","unclipped","continuous","actions","for","training",".","collect",":","If","True",",","creates","ops","for","actions_log_prob",",","value_preds",",","and","action_distribution_params",".","(","default","True",")","compute_value_and_advantage_in_train",":","A","bool","to","indicate","where","value","prediction","and","advantage","calculation","happen",".","If","True",",","both","happen","in","agent.train","(",")",",","therefore","no","need","to","save","the","value","prediction","inside","of","policy","info",".","If","False",",","value","prediction","is","computed","during","data","collection",".","This","argument","must","be","set","to","`","False","`","if","mini","batch","learning","is","enabled",".","Raises",":","TypeError",":","if","`","actor_network","`","or","`","value_network","`","is","not","of","type","`","tf_agents.networks.Network","`",".","ValueError",":","if","`","actor_network","`","or","`","value_network","`","do","not","emit","valid","outputs",".","For","example",",","`","actor_network","`","must","either","be","a","(","legacy","style",")","`","DistributionNetwork","`",",","or","explicitly","emit","a","nest","of","`","tfp.distribution.Distribution","`","objects","."]
code: def __init__(
      self,
      time_step_spec: ts.TimeStep,
      action_spec: types.NestedTensorSpec,
      actor_network: network.Network,
      value_network: network.Network,
      observation_normalizer: Optional[
          tensor_normalizer.TensorNormalizer
      ] = None,
      clip: bool = True,
      collect: bool = True,
      compute_value_and_advantage_in_train: bool = False,
  ):
      """Builds a PPO Policy given network Templates or functions.
  
      Args:
        time_step_spec: A `TimeStep` spec of the expected time_steps.
        action_spec: A nest of BoundedTensorSpec representing the actions.
        actor_network: An instance of a tf_agents.networks.network.Network, with
          call(observation, step_type, network_state).  Network should return one
          of the following: 1. a nested tuple of tfp.distributions objects
          matching action_spec, or 2. a nested tuple of tf.Tensors representing
          actions.
        value_network:  An instance of a tf_agents.networks.network.Network, with
          call(observation, step_type, network_state).  Network should return
          value predictions for the input state.
        observation_normalizer: An object to use for obervation normalization.
        clip: Whether to clip actions to spec before returning them.  Default
          True. Most policy-based algorithms (PCL, PPO, REINFORCE) use unclipped
          continuous actions for training.
        collect: If True, creates ops for actions_log_prob, value_preds, and
          action_distribution_params. (default True)
        compute_value_and_advantage_in_train: A bool to indicate where value
          prediction and advantage calculation happen.  If True, both happen in
          agent.train(), therefore no need to save the value prediction inside of
          policy info. If False, value prediction is computed during data
          collection. This argument must be set to `False` if mini batch learning
          is enabled.
  
      Raises:
        TypeError: if `actor_network` or `value_network` is not of type
          `tf_agents.networks.Network`.
        ValueError: if `actor_network` or `value_network` do not emit
          valid outputs.  For example, `actor_network` must either be
          a (legacy style) `DistributionNetwork`, or explicitly emit
          a nest of `tfp.distribution.Distribution` objects.
      """
      if not isinstance(actor_network, network.Network):
          raise TypeError(
              "actor_network is not of type network.Network"
          )
      if not isinstance(value_network, network.Network):
          raise TypeError(
              "value_network is not of type network.Network"
          )
  
      actor_output_spec = actor_network.create_variables(
          time_step_spec.observation
      )
  
      value_output_spec = value_network.create_variables(
          time_step_spec.observation
      )
  
      nest_utils.assert_value_spec(value_output_spec, "value_network")
  
      distribution_utils.assert_specs_are_compatible(
          actor_output_spec,
          action_spec,
          "actor_network output spec does not match action spec",
      )
  
      self._compute_value_and_advantage_in_train = (
          compute_value_and_advantage_in_train
      )
  
      if collect:
          if isinstance(actor_network, network.DistributionNetwork):
              # Legacy DistributionNetwork case.  New code can just provide a regular
              # Network that emits a Distribution object; and we use a different
              # code path using DistributionSpecV2 for that.
              network_output_spec = actor_network.output_spec
              info_spec = {
                  "dist_params": tf.nest.map_structure(
                      lambda spec: spec.input_params_spec,
                      network_output_spec,
                  )
              }
          else:
              # We have a Network that emits a nest of distributions.
              def nested_dist_params(spec):
                  if not isinstance(
                      spec, distribution_utils.DistributionSpecV2
                  ):
                      raise ValueError(
                          "Unexpected output from `actor_network`.  Expected "
                          "`Distribution` objects, but saw output spec: {}".format(
                              actor_output_spec
                          )
                      )
                  return distribution_utils.parameters_to_dict(
                      spec.parameters, tensors_only=True
                  )
  
              info_spec = {
                  "dist_params": tf.nest.map_structure(
                      nested_dist_params, actor_output_spec
                  )
              }
  
          if not self._compute_value_and_advantage_in_train:
              info_spec["value_prediction"] = tensor_spec.TensorSpec(
                  shape=[], dtype=tf.float32, name="value_prediction"
              )
      else:
          info_spec = ()
  
      policy_state_spec = {}
      if actor_network.state_spec:
          policy_state_spec[
              "actor_network_state"
          ] = actor_network.state_spec
      if (
          collect
          and value_network.state_spec
          and not self._compute_value_and_advantage_in_train
      ):
          policy_state_spec[
              "value_network_state"
          ] = value_network.state_spec
      if not policy_state_spec:
          policy_state_spec = ()
  
      super(PPOPolicy, self).__init__(
          time_step_spec=time_step_spec,
          action_spec=action_spec,
          policy_state_spec=policy_state_spec,
          info_spec=info_spec,
          actor_network=actor_network,
          observation_normalizer=observation_normalizer,
          clip=clip,
      )
  
      self._collect = collect
      self._value_network = value_network
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  from typing import Optional
  import gin
  import tensorflow
  from tf_agents.agents.ppo import ppo_utils
  from tf_agents.distributions import utils
  from tf_agents.networks import network
  from tf_agents.policies import actor_policy
  from tf_agents.specs import tensor_spec
  from tf_agents.trajectories import policy_step
  from tf_agents.trajectories import time_step
  from tf_agents.typing import types
  from tf_agents.utils import common
  from tf_agents.utils import nest_utils
  from tf_agents.utils import tensor_normalizer
label_desc: Use image node_id 1 to create a new PPOPolicy object from inherited base classes: actor_policy with example: obj = PPOPolicy(time_step_spec, action_spec, actor_network, value_network, observation_normalizer, clip, collect, compute_value_and_advantage_in_train)