class_docstr: Extension of the standard TorchModelV2 for SAC.
  
  To customize, do one of the following:
  - sub-class SACTorchModel and override one or more of its methods.
  - Use SAC's `q_model_config` and `policy_model` keys to tweak the default model
    behaviors (e.g. fcnet_hiddens, conv_filters, etc..).
  - Use SAC's `q_model_config->custom_model` and `policy_model->custom_model` keys
    to specify your own custom Q-model(s) and policy-models, which will be
    created within this SACTFModel (see `build_policy_model` and
    `build_q_model`.
  
  Note: It is not recommended to override the `forward` method for SAC. This
  would lead to shared weights (between policy and Q-nets), which will then
  not be optimized by either of the critic- or actor-optimizers!
  
  Data flow:
      `obs` -> forward() (should stay a noop method!) -> `model_out`
      `model_out` -> get_policy_output() -> pi(actions|obs)
      `model_out`, `actions` -> get_q_values() -> Q(s, a)
      `model_out`, `actions` -> get_twin_q_values() -> Q_twin(s, a)
class_docstr_tok: ["Extension","of","the","standard","TorchModelV2","for","SAC",".","To","customize",",","do","one","of","the","following",":","-","sub-class","SACTorchModel","and","override","one","or","more","of","its","methods",".","-","Use","SAC","'s","`","q_model_config","`","and","`","policy_model","`","keys","to","tweak","the","default","model","behaviors","(","e.g",".","fcnet_hiddens",",","conv_filters",",","etc","..",")",".","-","Use","SAC","'s","`","q_model_config-",">","custom_model","`","and","`","policy_model-",">","custom_model","`","keys","to","specify","your","own","custom","Q-model","(","s",")","and","policy-models",",","which","will","be","created","within","this","SACTFModel","(","see","`","build_policy_model","`","and","`","build_q_model","`",".","Note",":","It","is","not","recommended","to","override","the","`","forward","`","method","for","SAC",".","This","would","lead","to","shared","weights","(","between","policy","and","Q-nets",")",",","which","will","then","not","be","optimized","by","either","of","the","critic-","or","actor-optimizers","!","Data","flow",":","`","obs","`","-",">","forward","(",")","(","should","stay","a","noop","method","!",")","-",">","`","model_out","`","`","model_out","`","-",">","get_policy_output","(",")","-",">","pi","(","actions|obs",")","`","model_out","`",",","`","actions","`","-",">","get_q_values","(",")","-",">","Q","(","s",",","a",")","`","model_out","`",",","`","actions","`","-",">","get_twin_q_values","(",")","-",">","Q_twin","(","s",",","a",")"]
docstr: Initializes a SACTorchModel instance.
  7
          Args:
              policy_model_config: The config dict for the
                  policy network.
              q_model_config: The config dict for the
                  Q-network(s) (2 if twin_q=True).
              twin_q: Build twin Q networks (Q-net and target) for more
                  stable Q-learning.
              initial_alpha: The initial value for the to-be-optimized
                  alpha parameter (default: 1.0).
              target_entropy (Optional[float]): A target entropy value for
                  the to-be-optimized alpha parameter. If None, will use the
                  defaults described in the papers for SAC (and discrete SAC).
  
          Note that the core layers for forward() are not defined here, this
          only defines the layers for the output heads. Those layers for
          forward() should be defined in subclasses of SACModel.
docstr_tok: ["Initializes","a","SACTorchModel","instance",".","7","Args",":","policy_model_config",":","The","config","dict","for","the","policy","network",".","q_model_config",":","The","config","dict","for","the","Q-network","(","s",")","(","2","if","twin_q=True",")",".","twin_q",":","Build","twin","Q","networks","(","Q-net","and","target",")","for","more","stable","Q-learning",".","initial_alpha",":","The","initial","value","for","the","to-be-optimized","alpha","parameter","(","default",":","1.0",")",".","target_entropy","(","Optional","[","float","]",")",":","A","target","entropy","value","for","the","to-be-optimized","alpha","parameter",".","If","None",",","will","use","the","defaults","described","in","the","papers","for","SAC","(","and","discrete","SAC",")",".","Note","that","the","core","layers","for","forward","(",")","are","not","defined","here",",","this","only","defines","the","layers","for","the","output","heads",".","Those","layers","for","forward","(",")","should","be","defined","in","subclasses","of","SACModel","."]
code: def __init__(
      self,
      obs_space: gym.spaces.Space,
      action_space: gym.spaces.Space,
      num_outputs: Optional[int],
      model_config: ModelConfigDict,
      name: str,
      policy_model_config: ModelConfigDict = None,
      q_model_config: ModelConfigDict = None,
      twin_q: bool = False,
      initial_alpha: float = 1.0,
      target_entropy: Optional[float] = None,
  ):
      """Initializes a SACTorchModel instance.
      7
              Args:
                  policy_model_config: The config dict for the
                      policy network.
                  q_model_config: The config dict for the
                      Q-network(s) (2 if twin_q=True).
                  twin_q: Build twin Q networks (Q-net and target) for more
                      stable Q-learning.
                  initial_alpha: The initial value for the to-be-optimized
                      alpha parameter (default: 1.0).
                  target_entropy (Optional[float]): A target entropy value for
                      the to-be-optimized alpha parameter. If None, will use the
                      defaults described in the papers for SAC (and discrete SAC).
  
              Note that the core layers for forward() are not defined here, this
              only defines the layers for the output heads. Those layers for
              forward() should be defined in subclasses of SACModel.
      """
      nn.Module.__init__(self)
      super(SACTorchModel, self).__init__(
          obs_space, action_space, num_outputs, model_config, name
      )
  
      if isinstance(action_space, Discrete):
          self.action_dim = action_space.n
          self.discrete = True
          action_outs = q_outs = self.action_dim
      elif isinstance(action_space, Box):
          self.action_dim = np.product(action_space.shape)
          self.discrete = False
          action_outs = 2 * self.action_dim
          q_outs = 1
      else:
          assert isinstance(action_space, Simplex)
          self.action_dim = np.product(action_space.shape)
          self.discrete = False
          action_outs = self.action_dim
          q_outs = 1
  
      # Build the policy network.
      self.action_model = self.build_policy_model(
          self.obs_space,
          action_outs,
          policy_model_config,
          "policy_model",
      )
  
      # Build the Q-network(s).
      self.q_net = self.build_q_model(
          self.obs_space, self.action_space, q_outs, q_model_config, "q"
      )
      if twin_q:
          self.twin_q_net = self.build_q_model(
              self.obs_space,
              self.action_space,
              q_outs,
              q_model_config,
              "twin_q",
          )
      else:
          self.twin_q_net = None
  
      log_alpha = nn.Parameter(
          torch.from_numpy(np.array([np.log(initial_alpha)])).float()
      )
      self.register_parameter("log_alpha", log_alpha)
  
      # Auto-calculate the target entropy.
      if target_entropy is None or target_entropy == "auto":
          # See hyperparams in [2] (README.md).
          if self.discrete:
              target_entropy = 0.98 * np.array(
                  -np.log(1.0 / action_space.n), dtype=np.float32
              )
          # See [1] (README.md).
          else:
              target_entropy = -np.prod(action_space.shape)
  
      target_entropy = nn.Parameter(
          torch.from_numpy(np.array([target_entropy])).float(),
          requires_grad=False,
      )
      self.register_parameter("target_entropy", target_entropy)
  
imports: import gymnasium
  from gymnasium.spaces import Box, Discrete
  import numpy
  import tree
  from typing import Dict, List, Optional
  from ray.rllib.models.catalog import ModelCatalog
  from ray.rllib.models.torch.torch_modelv2 import TorchModelV2
  from ray.rllib.utils.annotations import override
  from ray.rllib.utils.framework import try_import_torch
  from ray.rllib.utils.spaces.simplex import Simplex
  from ray.rllib.utils.typing import ModelConfigDict, TensorType, TensorStructType
label_desc: Use image node_id 1 to create a new SACTorchModel object from inherited base classes: TorchModelV2, nn with example: obj = SACTorchModel(obs_space, action_space, num_outputs, model_config, name, policy_model_config, q_model_config, twin_q, initial_alpha, target_entropy)