class_docstr: :param num_steps_between_gradient_updates: (int)
      Every num_steps_between_gradient_updates transitions will be considered as a single batch and use for
      accumulating gradients. This is also the number of steps used for bootstrapping according to the n-step formulation.
  
  :param ratio_of_replay: (int)
      The number of off-policy training iterations in each ACER iteration.
  
  :param num_transitions_to_start_replay: (int)
      Number of environment steps until ACER starts to train off-policy from the experience replay.
      This emulates a heat-up phase where the agents learns only on-policy until there are enough transitions in
      the experience replay to start the off-policy training.
  
  :param rate_for_copying_weights_to_target: (float)
      The rate of the exponential moving average for the average policy which is used for the trust region optimization.
      The target network in this algorithm is used as the average policy.
  
  :param importance_weight_truncation: (float)
      The clipping constant for the importance weight truncation (not used in the Q-retrace calculation).
  
  :param use_trust_region_optimization: (bool)
      If set to True, the gradients of the network will be modified with a term dependant on the KL divergence between
      the average policy and the current one, to bound the change of the policy during the network update.
  
  :param max_KL_divergence: (float)
      The upper bound parameter for the trust region optimization, use_trust_region_optimization needs to be set true
      for this parameter to have an effect.
  
  :param beta_entropy: (float)
      An entropy regulaization term can be added to the loss function in order to control exploration. This term
      is weighted using the beta value defined by beta_entropy.
code: def __init__(self):
      super().__init__()
      self.apply_gradients_every_x_episodes = 5
      self.num_steps_between_gradient_updates = 5000
      self.ratio_of_replay = 4
      self.num_transitions_to_start_replay = 10000
      self.rate_for_copying_weights_to_target = 0.01
      self.importance_weight_truncation = 10.0
      self.use_trust_region_optimization = True
      self.max_KL_divergence = 1.0
      self.beta_entropy = 0
  
imports: from typing import Union
  import numpy
  from rl_coach.agents.policy_optimization_agent import PolicyOptimizationAgent
  from rl_coach.architectures.embedder_parameters import InputEmbedderParameters
  from rl_coach.architectures.head_parameters import ACERPolicyHeadParameters, QHeadParameters
  from rl_coach.architectures.middleware_parameters import FCMiddlewareParameters
  from rl_coach.base_parameters import AlgorithmParameters, NetworkParameters, AgentParameters
  from rl_coach.core_types import Batch
  from rl_coach.exploration_policies.categorical import CategoricalParameters
  from rl_coach.memories.episodic.episodic_experience_replay import EpisodicExperienceReplayParameters
  from rl_coach.spaces import DiscreteActionSpace
  from rl_coach.utils import eps, last_sample
label_desc: Use image node_id 1 to create a new ACERAlgorithmParameters object from inherited base classes: AlgorithmParameters with example: obj = ACERAlgorithmParameters()