class_docstr: A REINFORCE Agent.
  
  Implements:
  
  REINFORCE algorithm from
  
  "Simple statistical gradient-following algorithms for connectionist
  reinforcement learning"
  Williams, R.J., 1992.
  https://www-anw.cs.umass.edu/~barto/courses/cs687/williams92simple.pdf
  
  REINFORCE with state-value baseline, where state-values are estimated with
  function approximation, from
  
  "Reinforcement learning: An introduction" (Sec. 13.4)
  Sutton, R.S. and Barto, A.G., 2018.
  http://incompleteideas.net/book/the-book-2nd.html
  
  The REINFORCE agent can be optionally provided with:
  - value_network: A `tf_agents.network.Network` which parameterizes state-value
    estimation as a neural network. The network will be called with
    call(observation, step_type) and returns a floating point state-values
    tensor.
  - value_estimation_loss_coef: Weight on the value prediction loss.
  
  If value_network and value_estimation_loss_coef are provided, advantages are
  computed as
    `advantages = (discounted accumulated rewards) - (estimated state-values)`
  and the overall learning objective becomes:
    `(total loss) =
      (policy gradient loss) +
      value_estimation_loss_coef * (squared error of estimated state-values)`
class_docstr_tok: ["A","REINFORCE","Agent",".","Implements",":","REINFORCE","algorithm","from","''","Simple","statistical","gradient-following","algorithms","for","connectionist","reinforcement","learning","''","Williams",",","R.J.",",","1992.","https",":","\/\/www-anw.cs.umass.edu\/~barto\/courses\/cs687\/williams92simple.pdf","REINFORCE","with","state-value","baseline",",","where","state-values","are","estimated","with","function","approximation",",","from","''","Reinforcement","learning",":","An","introduction","''","(","Sec",".","13.4",")","Sutton",",","R.S",".","and","Barto",",","A.G.",",","2018.","http",":","\/\/incompleteideas.net\/book\/the-book-2nd.html","The","REINFORCE","agent","can","be","optionally","provided","with",":","-","value_network",":","A","`","tf_agents.network.Network","`","which","parameterizes","state-value","estimation","as","a","neural","network",".","The","network","will","be","called","with","call","(","observation",",","step_type",")","and","returns","a","floating","point","state-values","tensor",".","-","value_estimation_loss_coef",":","Weight","on","the","value","prediction","loss",".","If","value_network","and","value_estimation_loss_coef","are","provided",",","advantages","are","computed","as","`","advantages","=","(","discounted","accumulated","rewards",")","-","(","estimated","state-values",")","`","and","the","overall","learning","objective","becomes",":","`","(","total","loss",")","=","(","policy","gradient","loss",")","+","value_estimation_loss_coef","*","(","squared","error","of","estimated","state-values",")","`"]
docstr: Creates a REINFORCE Agent.
  
  Args:
    time_step_spec: A `TimeStep` spec of the expected time_steps.
    action_spec: A nest of BoundedTensorSpec representing the actions.
    actor_network: A tf_agents.network.Network to be used by the agent. The
      network will be called with call(observation, step_type).
    optimizer: Optimizer for the actor network.
    value_network: (Optional) A `tf_agents.network.Network` to be used by the
      agent. The network will be called with call(observation, step_type) and
      returns a floating point value tensor.
    value_estimation_loss_coef: (Optional) Multiplier for value prediction
      loss to balance with policy gradient loss.
    advantage_fn: A function `A(returns, value_preds)` that takes returns and
      value function predictions as input and returns advantages. The default
      is `A(returns, value_preds) = returns - value_preds` if a value network
      is specified and `use_advantage_loss=True`, otherwise `A(returns,
      value_preds) = returns`.
    use_advantage_loss: Whether to use value function predictions for
      computing returns. `use_advantage_loss=False` is equivalent to setting
      `advantage_fn=lambda returns, value_preds: returns`.
    gamma: A discount factor for future rewards.
    normalize_returns: Whether to normalize returns across episodes when
      computing the loss.
    gradient_clipping: Norm length to clip gradients.
    debug_summaries: A bool to gather debug summaries.
    summarize_grads_and_vars: If True, gradient and network variable summaries
      will be written during training.
    entropy_regularization: Coefficient for entropy regularization loss term.
    train_step_counter: An optional counter to increment every time the train
      op is run. Defaults to the global_step.
    name: The name of this agent. All variables in this module will fall under
      that name. Defaults to the class name.
docstr_tok: ["Creates","a","REINFORCE","Agent",".","Args",":","time_step_spec",":","A","`","TimeStep","`","spec","of","the","expected","time_steps",".","action_spec",":","A","nest","of","BoundedTensorSpec","representing","the","actions",".","actor_network",":","A","tf_agents.network.Network","to","be","used","by","the","agent",".","The","network","will","be","called","with","call","(","observation",",","step_type",")",".","optimizer",":","Optimizer","for","the","actor","network",".","value_network",":","(","Optional",")","A","`","tf_agents.network.Network","`","to","be","used","by","the","agent",".","The","network","will","be","called","with","call","(","observation",",","step_type",")","and","returns","a","floating","point","value","tensor",".","value_estimation_loss_coef",":","(","Optional",")","Multiplier","for","value","prediction","loss","to","balance","with","policy","gradient","loss",".","advantage_fn",":","A","function","`","A","(","returns",",","value_preds",")","`","that","takes","returns","and","value","function","predictions","as","input","and","returns","advantages",".","The","default","is","`","A","(","returns",",","value_preds",")","=","returns","-","value_preds","`","if","a","value","network","is","specified","and","`","use_advantage_loss=True","`",",","otherwise","`","A","(","returns",",","value_preds",")","=","returns","`",".","use_advantage_loss",":","Whether","to","use","value","function","predictions","for","computing","returns",".","`","use_advantage_loss=False","`","is","equivalent","to","setting","`","advantage_fn=lambda","returns",",","value_preds",":","returns","`",".","gamma",":","A","discount","factor","for","future","rewards",".","normalize_returns",":","Whether","to","normalize","returns","across","episodes","when","computing","the","loss",".","gradient_clipping",":","Norm","length","to","clip","gradients",".","debug_summaries",":","A","bool","to","gather","debug","summaries",".","summarize_grads_and_vars",":","If","True",",","gradient","and","network","variable","summaries","will","be","written","during","training",".","entropy_regularization",":","Coefficient","for","entropy","regularization","loss","term",".","train_step_counter",":","An","optional","counter","to","increment","every","time","the","train","op","is","run",".","Defaults","to","the","global_step",".","name",":","The","name","of","this","agent",".","All","variables","in","this","module","will","fall","under","that","name",".","Defaults","to","the","class","name","."]
code: def __init__(
      self,
      time_step_spec: ts.TimeStep,
      action_spec: types.TensorSpec,
      actor_network: network.Network,
      optimizer: types.Optimizer,
      value_network: Optional[network.Network] = None,
      value_estimation_loss_coef: types.Float = 0.2,
      advantage_fn: Optional[AdvantageFnType] = None,
      use_advantage_loss: bool = True,
      gamma: types.Float = 1.0,
      normalize_returns: bool = True,
      gradient_clipping: Optional[types.Float] = None,
      debug_summaries: bool = False,
      summarize_grads_and_vars: bool = False,
      entropy_regularization: Optional[types.Float] = None,
      train_step_counter: Optional[tf.Variable] = None,
      name: Optional[Text] = None,
  ):
      """Creates a REINFORCE Agent.
  
      Args:
        time_step_spec: A `TimeStep` spec of the expected time_steps.
        action_spec: A nest of BoundedTensorSpec representing the actions.
        actor_network: A tf_agents.network.Network to be used by the agent. The
          network will be called with call(observation, step_type).
        optimizer: Optimizer for the actor network.
        value_network: (Optional) A `tf_agents.network.Network` to be used by the
          agent. The network will be called with call(observation, step_type) and
          returns a floating point value tensor.
        value_estimation_loss_coef: (Optional) Multiplier for value prediction
          loss to balance with policy gradient loss.
        advantage_fn: A function `A(returns, value_preds)` that takes returns and
          value function predictions as input and returns advantages. The default
          is `A(returns, value_preds) = returns - value_preds` if a value network
          is specified and `use_advantage_loss=True`, otherwise `A(returns,
          value_preds) = returns`.
        use_advantage_loss: Whether to use value function predictions for
          computing returns. `use_advantage_loss=False` is equivalent to setting
          `advantage_fn=lambda returns, value_preds: returns`.
        gamma: A discount factor for future rewards.
        normalize_returns: Whether to normalize returns across episodes when
          computing the loss.
        gradient_clipping: Norm length to clip gradients.
        debug_summaries: A bool to gather debug summaries.
        summarize_grads_and_vars: If True, gradient and network variable summaries
          will be written during training.
        entropy_regularization: Coefficient for entropy regularization loss term.
        train_step_counter: An optional counter to increment every time the train
          op is run. Defaults to the global_step.
        name: The name of this agent. All variables in this module will fall under
          that name. Defaults to the class name.
      """
      tf.Module.__init__(self, name=name)
  
      actor_network.create_variables()
      self._actor_network = actor_network
      if value_network:
          value_network.create_variables()
      self._value_network = value_network
  
      collect_policy = actor_policy.ActorPolicy(
          time_step_spec=time_step_spec,
          action_spec=action_spec,
          actor_network=self._actor_network,
          clip=True,
      )
  
      policy = greedy_policy.GreedyPolicy(collect_policy)
  
      self._optimizer = optimizer
      self._gamma = gamma
      self._normalize_returns = normalize_returns
      self._gradient_clipping = gradient_clipping
      self._entropy_regularization = entropy_regularization
      self._value_estimation_loss_coef = value_estimation_loss_coef
      self._baseline = self._value_network is not None
      self._advantage_fn = advantage_fn
      if self._advantage_fn is None:
          if use_advantage_loss and self._baseline:
              self._advantage_fn = (
                  lambda returns, value_preds: returns - value_preds
              )
          else:
              self._advantage_fn = lambda returns, _: returns
  
      super(ReinforceAgent, self).__init__(
          time_step_spec,
          action_spec,
          policy,
          collect_policy,
          train_sequence_length=None,
          debug_summaries=debug_summaries,
          summarize_grads_and_vars=summarize_grads_and_vars,
          train_step_counter=train_step_counter,
      )
      self._as_trajectory = data_converter.AsTrajectory(
          self.data_context
      )
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  import collections
  from typing import Callable, Optional, Text
  import gin
  import numpy
  import tensorflow
  from tf_agents.agents import data_converter
  from tf_agents.agents import tf_agent
  from tf_agents.networks import network
  from tf_agents.policies import actor_policy
  from tf_agents.policies import greedy_policy
  from tf_agents.trajectories import time_step
  from tf_agents.trajectories import trajectory
  from tf_agents.typing import types
  from tf_agents.utils import common
  from tf_agents.utils import eager_utils
  from tf_agents.utils import nest_utils
  from tf_agents.utils import value_ops
label_desc: Use image node_id 1 to create a new ReinforceAgent object from inherited base classes: tf_agent with example: obj = ReinforceAgent(time_step_spec, action_spec, actor_network, optimizer, value_network, value_estimation_loss_coef, advantage_fn, use_advantage_loss, gamma, normalize_returns, gradient_clipping, debug_summaries, summarize_grads_and_vars, entropy_regularization, train_step_counter, name)