class_docstr: A SAC Model in Torch.
  
  Based on Soft Actor-Critic and Applications:
      https://arxiv.org/abs/1812.05905
  
  Soft Actor-Critic (SAC) is an algorithm which optimizes a stochastic
  policy in an off-policy way, forming a bridge between stochastic policy
  optimization and DDPG-style approaches.
  A central feature of SAC is entropy regularization. The policy is trained
  to maximize a trade-off between expected return and entropy, a measure of
  randomness in the policy. This has a close connection to the
  exploration-exploitation trade-off: increasing entropy results in more
  exploration, which can accelerate learning later on. It can also prevent
  the policy from prematurely converging to a bad local optimum.
  
  Args:
      policy (garage.torch.policy.Policy): Policy/Actor/Agent that is being
          optimized by SAC.
      qf1 (garage.torch.q_function.ContinuousMLPQFunction): QFunction/Critic
          used for actor/policy optimization. See Soft Actor-Critic and
          Applications.
      qf2 (garage.torch.q_function.ContinuousMLPQFunction): QFunction/Critic
          used for actor/policy optimization. See Soft Actor-Critic and
          Applications.
      replay_buffer (ReplayBuffer): Stores transitions that are previously
          collected by the sampler.
      sampler (garage.sampler.Sampler): Sampler.
      env_spec (EnvSpec): The env_spec attribute of the environment that the
          agent is being trained in.
      max_episode_length_eval (int or None): Maximum length of episodes used
          for off-policy evaluation. If None, defaults to
          `env_spec.max_episode_length`.
      gradient_steps_per_itr (int): Number of optimization steps that should
      gradient_steps_per_itr(int): Number of optimization steps that should
          occur before the training step is over and a new batch of
          transitions is collected by the sampler.
      fixed_alpha (float): The entropy/temperature to be used if temperature
          is not supposed to be learned.
      target_entropy (float): target entropy to be used during
          entropy/temperature optimization. If None, the default heuristic
          from Soft Actor-Critic Algorithms and Applications is used.
      initial_log_entropy (float): initial entropy/temperature coefficient
          to be used if a fixed_alpha is not being used (fixed_alpha=None),
          and the entropy/temperature coefficient is being learned.
      discount (float): Discount factor to be used during sampling and
          critic/q_function optimization.
      buffer_batch_size (int): The number of transitions sampled from the
          replay buffer that are used during a single optimization step.
      min_buffer_size (int): The minimum number of transitions that need to
          be in the replay buffer before training can begin.
      target_update_tau (float): coefficient that controls the rate at which
          the target q_functions update over optimization iterations.
      policy_lr (float): learning rate for policy optimizers.
      qf_lr (float): learning rate for q_function optimizers.
      reward_scale (float): reward scale. Changing this hyperparameter
          changes the effect that the reward from a transition will have
          during optimization.
      optimizer (torch.optim.Optimizer): optimizer to be used for
          policy/actor, q_functions/critics, and temperature/entropy
          optimizations.
      steps_per_epoch (int): Number of train_once calls per epoch.
      num_evaluation_episodes (int): The number of evaluation episodes used
          for computing eval stats at the end of every epoch.
      eval_env (Environment): environment used for collecting evaluation
          episodes. If None, a copy of the train env is used.
      use_deterministic_evaluation (bool): True if the trained policy
          should be evaluated deterministically.
      temporal_regularization_factor (float): coefficient that determines
          the temporal regularization penalty as defined in CAPS as lambda_t
      spatial_regularization_factor (float): coefficient that determines
          the spatial regularization penalty as defined in CAPS as lambda_s
      spatial_regularization_eps (float): sigma of the normal distribution
          from with spatial regularization observations are drawn,
          in caps this is defined as epsilon_s
class_docstr_tok: ["A","SAC","Model","in","Torch",".","Based","on","Soft","Actor-Critic","and","Applications",":","https",":","\/\/arxiv.org\/abs\/1812.05905","Soft","Actor-Critic","(","SAC",")","is","an","algorithm","which","optimizes","a","stochastic","policy","in","an","off-policy","way",",","forming","a","bridge","between","stochastic","policy","optimization","and","DDPG-style","approaches",".","A","central","feature","of","SAC","is","entropy","regularization",".","The","policy","is","trained","to","maximize","a","trade-off","between","expected","return","and","entropy",",","a","measure","of","randomness","in","the","policy",".","This","has","a","close","connection","to","the","exploration-exploitation","trade-off",":","increasing","entropy","results","in","more","exploration",",","which","can","accelerate","learning","later","on",".","It","can","also","prevent","the","policy","from","prematurely","converging","to","a","bad","local","optimum",".","Args",":","policy","(","garage.torch.policy.Policy",")",":","Policy\/Actor\/Agent","that","is","being","optimized","by","SAC",".","qf1","(","garage.torch.q_function.ContinuousMLPQFunction",")",":","QFunction\/Critic","used","for","actor\/policy","optimization",".","See","Soft","Actor-Critic","and","Applications",".","qf2","(","garage.torch.q_function.ContinuousMLPQFunction",")",":","QFunction\/Critic","used","for","actor\/policy","optimization",".","See","Soft","Actor-Critic","and","Applications",".","replay_buffer","(","ReplayBuffer",")",":","Stores","transitions","that","are","previously","collected","by","the","sampler",".","sampler","(","garage.sampler.Sampler",")",":","Sampler",".","env_spec","(","EnvSpec",")",":","The","env_spec","attribute","of","the","environment","that","the","agent","is","being","trained","in",".","max_episode_length_eval","(","int","or","None",")",":","Maximum","length","of","episodes","used","for","off-policy","evaluation",".","If","None",",","defaults","to","`","env_spec.max_episode_length","`",".","gradient_steps_per_itr","(","int",")",":","Number","of","optimization","steps","that","should","gradient_steps_per_itr","(","int",")",":","Number","of","optimization","steps","that","should","occur","before","the","training","step","is","over","and","a","new","batch","of","transitions","is","collected","by","the","sampler",".","fixed_alpha","(","float",")",":","The","entropy\/temperature","to","be","used","if","temperature","is","not","supposed","to","be","learned",".","target_entropy","(","float",")",":","target","entropy","to","be","used","during","entropy\/temperature","optimization",".","If","None",",","the","default","heuristic","from","Soft","Actor-Critic","Algorithms","and","Applications","is","used",".","initial_log_entropy","(","float",")",":","initial","entropy\/temperature","coefficient","to","be","used","if","a","fixed_alpha","is","not","being","used","(","fixed_alpha=None",")",",","and","the","entropy\/temperature","coefficient","is","being","learned",".","discount","(","float",")",":","Discount","factor","to","be","used","during","sampling","and","critic\/q_function","optimization",".","buffer_batch_size","(","int",")",":","The","number","of","transitions","sampled","from","the","replay","buffer","that","are","used","during","a","single","optimization","step",".","min_buffer_size","(","int",")",":","The","minimum","number","of","transitions","that","need","to","be","in","the","replay","buffer","before","training","can","begin",".","target_update_tau","(","float",")",":","coefficient","that","controls","the","rate","at","which","the","target","q_functions","update","over","optimization","iterations",".","policy_lr","(","float",")",":","learning","rate","for","policy","optimizers",".","qf_lr","(","float",")",":","learning","rate","for","q_function","optimizers",".","reward_scale","(","float",")",":","reward","scale",".","Changing","this","hyperparameter","changes","the","effect","that","the","reward","from","a","transition","will","have","during","optimization",".","optimizer","(","torch.optim.Optimizer",")",":","optimizer","to","be","used","for","policy\/actor",",","q_functions\/critics",",","and","temperature\/entropy","optimizations",".","steps_per_epoch","(","int",")",":","Number","of","train_once","calls","per","epoch",".","num_evaluation_episodes","(","int",")",":","The","number","of","evaluation","episodes","used","for","computing","eval","stats","at","the","end","of","every","epoch",".","eval_env","(","Environment",")",":","environment","used","for","collecting","evaluation","episodes",".","If","None",",","a","copy","of","the","train","env","is","used",".","use_deterministic_evaluation","(","bool",")",":","True","if","the","trained","policy","should","be","evaluated","deterministically",".","temporal_regularization_factor","(","float",")",":","coefficient","that","determines","the","temporal","regularization","penalty","as","defined","in","CAPS","as","lambda_t","spatial_regularization_factor","(","float",")",":","coefficient","that","determines","the","spatial","regularization","penalty","as","defined","in","CAPS","as","lambda_s","spatial_regularization_eps","(","float",")",":","sigma","of","the","normal","distribution","from","with","spatial","regularization","observations","are","drawn",",","in","caps","this","is","defined","as","epsilon_s"]
docstr: Compute the Q-function/critic loss.
  
  Args:
      samples_data (dict): Transitions(S,A,R,S') that are sampled from
          the replay buffer. It should have the keys 'observation',
          'action', 'reward', 'terminal', and 'next_observations'.
  
  Note:
      samples_data's entries should be torch.Tensor's with the following
      shapes:
          observation: :math:`(N, O^*)`
          action: :math:`(N, A^*)`
          reward: :math:`(N, 1)`
          terminal: :math:`(N, 1)`
          next_observation: :math:`(N, O^*)`
  
  Returns:
      torch.Tensor: loss from 1st q-function after optimization.
      torch.Tensor: loss from 2nd q-function after optimization.
docstr_tok: ["Compute","the","Q-function\/critic","loss",".","Args",":","samples_data","(","dict",")",":","Transitions","(","S",",","A",",","R",",","S","'",")","that","are","sampled","from","the","replay","buffer",".","It","should","have","the","keys","'observation","'",",","'action","'",",","'reward","'",",","'terminal","'",",","and","'next_observations","'",".","Note",":","samples_data","'s","entries","should","be","torch.Tensor","'s","with","the","following","shapes",":","observation",":",":","math",":","`","(","N",",","O^","*",")","`","action",":",":","math",":","`","(","N",",","A^","*",")","`","reward",":",":","math",":","`","(","N",",","1",")","`","terminal",":",":","math",":","`","(","N",",","1",")","`","next_observation",":",":","math",":","`","(","N",",","O^","*",")","`","Returns",":","torch.Tensor",":","loss","from","1st","q-function","after","optimization",".","torch.Tensor",":","loss","from","2nd","q-function","after","optimization","."]
code: def _critic_objective(self, samples_data):
      """Compute the Q-function/critic loss.
  
      Args:
          samples_data (dict): Transitions(S,A,R,S') that are sampled from
              the replay buffer. It should have the keys 'observation',
              'action', 'reward', 'terminal', and 'next_observations'.
  
      Note:
          samples_data's entries should be torch.Tensor's with the following
          shapes:
              observation: :math:`(N, O^*)`
              action: :math:`(N, A^*)`
              reward: :math:`(N, 1)`
              terminal: :math:`(N, 1)`
              next_observation: :math:`(N, O^*)`
  
      Returns:
          torch.Tensor: loss from 1st q-function after optimization.
          torch.Tensor: loss from 2nd q-function after optimization.
  
      """
      obs = samples_data["observation"]
      actions = samples_data["action"]
      rewards = samples_data["reward"].flatten()
      terminals = samples_data["terminal"].flatten()
      next_obs = samples_data["next_observation"]
      with torch.no_grad():
          alpha = self._get_log_alpha(samples_data).exp()
  
      q1_pred = self._qf1(obs, actions)
      q2_pred = self._qf2(obs, actions)
  
      new_next_actions_dist = self.policy(next_obs)[0]
      (
          new_next_actions_pre_tanh,
          new_next_actions,
      ) = new_next_actions_dist.rsample_with_pre_tanh_value()
      new_log_pi = new_next_actions_dist.log_prob(
          value=new_next_actions,
          pre_tanh_value=new_next_actions_pre_tanh,
      )
  
      target_q_values = torch.min(
          self._target_qf1(next_obs, new_next_actions),
          self._target_qf2(next_obs, new_next_actions),
      ).flatten() - (alpha * new_log_pi)
      with torch.no_grad():
          q_target = (
              rewards * self._reward_scale
              + (1.0 - terminals) * self._discount * target_q_values
          )
      qf1_loss = F.mse_loss(q1_pred.flatten(), q_target)
      qf2_loss = F.mse_loss(q2_pred.flatten(), q_target)
  
      return qf1_loss, qf2_loss
  
imports: from collections import deque
  import copy
  from dowel import tabular
  import numpy
  import torch
  import torch.nn.functional
  from garage import log_performance, obtain_evaluation_episodes, StepType
  from garage.np.algos import RLAlgorithm
  from garage.torch import as_torch_dict, global_device, state_dict_to
  from garage.torch._functions import list_to_tensor, zero_optim_grads
label_desc: Use image node_id 7 for calling the SAC obj's underlying member method code with example usage: obj._critic_objective(samples_data) and returns: qf1_loss,  qf2_loss