class_docstr: :param policy_gradient_rescaler: (PolicyGradientRescaler)
      This represents how the critic will be used to update the actor. The critic value function is typically used
      to rescale the gradients calculated by the actor. There are several ways for doing this, such as using the
      advantage of the action, or the generalized advantage estimation (GAE) value.
  
  :param gae_lambda: (float)
      The :math:`\lambda` value is used within the GAE function in order to weight different bootstrap length
      estimations. Typical values are in the range 0.9-1, and define an exponential decay over the different
      n-step estimations.
  
  :param clip_likelihood_ratio_using_epsilon: (float)
      If not None, the likelihood ratio between the current and new policy in the PPO loss function will be
      clipped to the range [1-clip_likelihood_ratio_using_epsilon, 1+clip_likelihood_ratio_using_epsilon].
      This is typically used in the Clipped PPO version of PPO, and should be set to None in regular PPO
      implementations.
  
  :param value_targets_mix_fraction: (float)
      The targets for the value network are an exponential weighted moving average which uses this mix fraction to
      define how much of the new targets will be taken into account when calculating the loss.
      This value should be set to the range (0,1], where 1 means that only the new targets will be taken into account.
  
  :param estimate_state_value_using_gae: (bool)
      If set to True, the state value will be estimated using the GAE technique.
  
  :param use_kl_regularization: (bool)
      If set to True, the loss function will be regularized using the KL diveregence between the current and new
      policy, to bound the change of the policy during the network update.
  
  :param beta_entropy: (float)
      An entropy regulaization term can be added to the loss function in order to control exploration. This term
      is weighted using the :math:`eta` value defined by beta_entropy.
  
  :param optimization_epochs: (int)
      For each training phase, the collected dataset will be used for multiple epochs, which are defined by the
      optimization_epochs value.
  
  :param optimization_epochs: (Schedule)
      Can be used to define a schedule over the clipping of the likelihood ratio.
class_docstr_tok: [":","param","policy_gradient_rescaler",":","(","PolicyGradientRescaler",")","This","represents","how","the","critic","will","be","used","to","update","the","actor",".","The","critic","value","function","is","typically","used","to","rescale","the","gradients","calculated","by","the","actor",".","There","are","several","ways","for","doing","this",",","such","as","using","the","advantage","of","the","action",",","or","the","generalized","advantage","estimation","(","GAE",")","value",".",":","param","gae_lambda",":","(","float",")","The",":","math",":","`","\\lambda","`","value","is","used","within","the","GAE","function","in","order","to","weight","different","bootstrap","length","estimations",".","Typical","values","are","in","the","range","0.9-1",",","and","define","an","exponential","decay","over","the","different","n-step","estimations",".",":","param","clip_likelihood_ratio_using_epsilon",":","(","float",")","If","not","None",",","the","likelihood","ratio","between","the","current","and","new","policy","in","the","PPO","loss","function","will","be","clipped","to","the","range","[","1-clip_likelihood_ratio_using_epsilon",",","1+clip_likelihood_ratio_using_epsilon","]",".","This","is","typically","used","in","the","Clipped","PPO","version","of","PPO",",","and","should","be","set","to","None","in","regular","PPO","implementations",".",":","param","value_targets_mix_fraction",":","(","float",")","The","targets","for","the","value","network","are","an","exponential","weighted","moving","average","which","uses","this","mix","fraction","to","define","how","much","of","the","new","targets","will","be","taken","into","account","when","calculating","the","loss",".","This","value","should","be","set","to","the","range","(","0,1","]",",","where","1","means","that","only","the","new","targets","will","be","taken","into","account",".",":","param","estimate_state_value_using_gae",":","(","bool",")","If","set","to","True",",","the","state","value","will","be","estimated","using","the","GAE","technique",".",":","param","use_kl_regularization",":","(","bool",")","If","set","to","True",",","the","loss","function","will","be","regularized","using","the","KL","diveregence","between","the","current","and","new","policy",",","to","bound","the","change","of","the","policy","during","the","network","update",".",":","param","beta_entropy",":","(","float",")","An","entropy","regulaization","term","can","be","added","to","the","loss","function","in","order","to","control","exploration",".","This","term","is","weighted","using","the",":","math",":","`","\beta","`","value","defined","by","beta_entropy",".",":","param","optimization_epochs",":","(","int",")","For","each","training","phase",",","the","collected","dataset","will","be","used","for","multiple","epochs",",","which","are","defined","by","the","optimization_epochs","value",".",":","param","optimization_epochs",":","(","Schedule",")","Can","be","used","to","define","a","schedule","over","the","clipping","of","the","likelihood","ratio","."]
code: def __init__(self):
      super().__init__()
      self.num_episodes_in_experience_replay = 1000000
      self.policy_gradient_rescaler = PolicyGradientRescaler.GAE
      self.gae_lambda = 0.95
      self.use_kl_regularization = False
      self.clip_likelihood_ratio_using_epsilon = 0.2
      self.estimate_state_value_using_gae = True
      self.beta_entropy = 0.01  # should be 0 for mujoco
      self.num_consecutive_playing_steps = EnvironmentSteps(2048)
      self.optimization_epochs = 10
      self.normalization_stats = None
      self.clipping_decay_schedule = ConstantSchedule(1)
      self.act_for_full_episodes = True
      self.update_pre_network_filters_state_on_train = True
      self.update_pre_network_filters_state_on_inference = False
  
imports: import copy
  import math
  from collections import OrderedDict
  from random import shuffle
  from typing import Union
  import numpy
  from rl_coach.agents.actor_critic_agent import ActorCriticAgent
  from rl_coach.agents.policy_optimization_agent import PolicyGradientRescaler
  from rl_coach.architectures.embedder_parameters import InputEmbedderParameters
  from rl_coach.architectures.head_parameters import PPOHeadParameters, VHeadParameters
  from rl_coach.architectures.middleware_parameters import FCMiddlewareParameters
  from rl_coach.base_parameters import AlgorithmParameters, NetworkParameters, AgentParameters
  from rl_coach.core_types import EnvironmentSteps, Batch, EnvResponse, StateType
  from rl_coach.exploration_policies.additive_noise import AdditiveNoiseParameters
  from rl_coach.exploration_policies.categorical import CategoricalParameters
  from rl_coach.logger import screen
  from rl_coach.memories.episodic.episodic_experience_replay import EpisodicExperienceReplayParameters
  from rl_coach.schedules import ConstantSchedule
  from rl_coach.spaces import DiscreteActionSpace, BoxActionSpace
label_desc: Use image node_id 1 to create a new ClippedPPOAlgorithmParameters object from inherited base classes: AlgorithmParameters with example: obj = ClippedPPOAlgorithmParameters()