class_docstr: Policy which parametrizes independent action-/advantage-/state-value functions per action and
  optionally a state-value function, conditioned on the output of a central neural network
  processing the input state (specification key: `parametrized_value_policy`).
  
  Args:
      network ('auto' | specification): Policy network configuration, see
          [networks](../modules/networks.html)
          (<span style="color:#00C000"><b>default</b></span>: 'auto', automatically configured
          network).
      single_output (bool): Whether the network returns a single embedding tensor or, in the case
          of multiple action components, specifies additional outputs for some/all action/state
          value functions, via registered tensors with name "[ACTION]-embedding" or
          "state-embedding"/"[ACTION]-state-embedding" depending on the state_value_mode argument
          (<span style="color:#00C000"><b>default</b></span>: single output).
      state_value_mode ('implicit' | 'separate' | 'separate-per-action'): Whether to compute the
          state value implicitly as maximum action value (like DQN), or as either a single
          separate state-value function or a function per action (like DuelingDQN)
          (<span style="color:#00C000"><b>default</b></span>: single separate state-value
          function).
      device (string): Device name
          (<span style="color:#00C000"><b>default</b></span>: inherit value of parent module).
      l2_regularization (float >= 0.0): Scalar controlling L2 regularization
          (<span style="color:#00C000"><b>default</b></span>: inherit value of parent module).
      name (string): <span style="color:#0000C0"><b>internal use</b></span>.
      states_spec (specification): <span style="color:#0000C0"><b>internal use</b></span>.
      auxiliaries_spec (specification): <span style="color:#0000C0"><b>internal use</b></span>.
      internals_spec (specification): <span style="color:#0000C0"><b>internal use</b></span>.
      actions_spec (specification): <span style="color:#0000C0"><b>internal use</b></span>.
class_docstr_tok: ["Policy","which","parametrizes","independent","action-\/advantage-\/state-value","functions","per","action","and","optionally","a","state-value","function",",","conditioned","on","the","output","of","a","central","neural","network","processing","the","input","state","(","specification","key",":","`","parametrized_value_policy","`",")",".","Args",":","network","(","'auto","'","|","specification",")",":","Policy","network","configuration",",","see","[","networks","]","(","..","\/modules\/networks.html",")","(","<","span","style=","''","color",":","#","00C000","''",">","<","b",">","default","<","\/b",">","<","\/span",">",":","'auto","'",",","automatically","configured","network",")",".","single_output","(","bool",")",":","Whether","the","network","returns","a","single","embedding","tensor","or",",","in","the","case","of","multiple","action","components",",","specifies","additional","outputs","for","some\/all","action\/state","value","functions",",","via","registered","tensors","with","name","``","[","ACTION","]","-embedding","''","or","``","state-embedding","''","\/","''","[","ACTION","]","-state-embedding","''","depending","on","the","state_value_mode","argument","(","<","span","style=","''","color",":","#","00C000","''",">","<","b",">","default","<","\/b",">","<","\/span",">",":","single","output",")",".","state_value_mode","(","'implicit","'","|","'separate","'","|","'separate-per-action","'",")",":","Whether","to","compute","the","state","value","implicitly","as","maximum","action","value","(","like","DQN",")",",","or","as","either","a","single","separate","state-value","function","or","a","function","per","action","(","like","DuelingDQN",")","(","<","span","style=","''","color",":","#","00C000","''",">","<","b",">","default","<","\/b",">","<","\/span",">",":","single","separate","state-value","function",")",".","device","(","string",")",":","Device","name","(","<","span","style=","''","color",":","#","00C000","''",">","<","b",">","default","<","\/b",">","<","\/span",">",":","inherit","value","of","parent","module",")",".","l2_regularization","(","float",">","=","0.0",")",":","Scalar","controlling","L2","regularization","(","<","span","style=","''","color",":","#","00C000","''",">","<","b",">","default","<","\/b",">","<","\/span",">",":","inherit","value","of","parent","module",")",".","name","(","string",")",":","<","span","style=","''","color",":","#","0000C0","''",">","<","b",">","internal","use","<","\/b",">","<","\/span",">",".","states_spec","(","specification",")",":","<","span","style=","''","color",":","#","0000C0","''",">","<","b",">","internal","use","<","\/b",">","<","\/span",">",".","auxiliaries_spec","(","specification",")",":","<","span","style=","''","color",":","#","0000C0","''",">","<","b",">","internal","use","<","\/b",">","<","\/span",">",".","internals_spec","(","specification",")",":","<","span","style=","''","color",":","#","0000C0","''",">","<","b",">","internal","use","<","\/b",">","<","\/span",">",".","actions_spec","(","specification",")",":","<","span","style=","''","color",":","#","0000C0","''",">","<","b",">","internal","use","<","\/b",">","<","\/span",">","."]
code: def action_values(
      self, *, states, horizons, internals, auxiliaries, actions
  ):
      deterministic = tf_util.constant(value=True, dtype="bool")
      embedding, _ = self.network.apply(
          x=states,
          horizons=horizons,
          internals=internals,
          deterministic=deterministic,
          independent=True,
      )
      if not isinstance(embedding, TensorDict):
          embedding = TensorDict(embedding=embedding)
  
      if self.state_value_mode == "implicit":
  
          def function(name, spec, a_value, action):
              if name is None:
                  x = embedding.get(
                      "action-embedding", embedding["embedding"]
                  )
              else:
                  x = embedding.get(
                      name + "-embedding", embedding["embedding"]
                  )
              action_value = a_value.apply(x=x)
              if spec.type == "bool":
                  shape = (-1,) + spec.shape + (2,)
              elif spec.type == "int":
                  shape = (-1,) + spec.shape + (spec.num_values,)
              action_value = tf.reshape(
                  tensor=action_value, shape=shape
              )
              if spec.type == "bool":
                  return tf.where(
                      condition=action,
                      x=action_value[..., 0],
                      y=action_value[..., 1],
                  )
              elif spec.type == "int":
                  action = tf.expand_dims(
                      input=action, axis=(spec.rank + 1)
                  )
                  action_value = tf.gather(
                      params=action_value,
                      indices=action,
                      batch_dims=(spec.rank + 1),
                  )
                  return tf.squeeze(
                      input=action_value, axis=(spec.rank + 1)
                  )
  
          return self.actions_spec.fmap(
              function=function,
              cls=TensorDict,
              with_names=True,
              zip_values=(self.a_values, actions),
          )
  
      elif self.state_value_mode == "separate":
          state_value = self.s_value.apply(
              x=embedding.get("state-embedding", embedding["embedding"])
          )
  
          def function(name, spec, a_value, action):
              if name is None:
                  x = embedding.get(
                      "action-embedding", embedding["embedding"]
                  )
              else:
                  x = embedding.get(
                      name + "-embedding", embedding["embedding"]
                  )
              advantage_value = a_value.apply(x=x)
              if spec.type == "bool":
                  shape = (-1,) + spec.shape + (2,)
              elif spec.type == "int":
                  shape = (-1,) + spec.shape + (spec.num_values,)
              advantage_value = tf.reshape(
                  tensor=advantage_value, shape=shape
              )
              mean = tf.math.reduce_mean(
                  input_tensor=advantage_value, axis=-1, keepdims=True
              )
              shape = (-1,) + tuple(1 for _ in range(spec.rank + 1))
              _state_value = tf.reshape(tensor=state_value, shape=shape)
              action_value = _state_value + (advantage_value - mean)
              if spec.type == "bool":
                  return tf.where(
                      condition=action,
                      x=action_value[..., 0],
                      y=action_value[..., 1],
                  )
              elif spec.type == "int":
                  action = tf.expand_dims(
                      input=action, axis=(spec.rank + 1)
                  )
                  action_value = tf.gather(
                      params=action_value,
                      indices=action,
                      batch_dims=(spec.rank + 1),
                  )
                  return tf.squeeze(
                      input=action_value, axis=(spec.rank + 1)
                  )
  
          return self.actions_spec.fmap(
              function=function,
              cls=TensorDict,
              with_names=True,
              zip_values=(self.a_values, actions),
          )
  
      elif self.state_value_mode == "separate-per-action":
  
          def function(name, spec, s_value, a_value, action):
              if name is None:
                  state_value = s_value.apply(
                      x=embedding.get(
                          "state-embedding", embedding["embedding"]
                      )
                  )
                  advantage_value = a_value.apply(
                      x=embedding.get(
                          "action-embedding", embedding["embedding"]
                      )
                  )
              else:
                  state_value = s_value.apply(
                      x=embedding.get(
                          name + "-state-embedding",
                          embedding["embedding"],
                      )
                  )
                  advantage_value = a_value.apply(
                      x=embedding.get(
                          name + "-embedding", embedding["embedding"]
                      )
                  )
              if spec.type == "bool":
                  shape = (-1,) + spec.shape + (2,)
              elif spec.type == "int":
                  shape = (-1,) + spec.shape + (spec.num_values,)
              advantage_value = tf.reshape(
                  tensor=advantage_value, shape=shape
              )
              mean = tf.math.reduce_mean(
                  input_tensor=advantage_value, axis=-1, keepdims=True
              )
              action_value = tf.expand_dims(
                  input=state_value, axis=-1
              ) + (advantage_value - mean)
              if spec.type == "bool":
                  return tf.where(
                      condition=action,
                      x=action_value[..., 0],
                      y=action_value[..., 1],
                  )
              elif spec.type == "int":
                  action = tf.expand_dims(
                      input=action, axis=(spec.rank + 1)
                  )
                  action_value = tf.gather(
                      params=action_value,
                      indices=action,
                      batch_dims=(spec.rank + 1),
                  )
                  return tf.squeeze(
                      input=action_value, axis=(spec.rank + 1)
                  )
  
          return self.actions_spec.fmap(
              function=function,
              cls=TensorDict,
              with_names=True,
              zip_values=(self.s_values, self.a_values, actions),
          )
  
file_path: tensorforce/tensorforce/core/policies/parametrized_value_policy.py
imports: import tensorflow
  from tensorforce import TensorforceError
  from tensorforce.core import layer_modules, ModuleDict, TensorDict, TensorSpec, TensorsSpec, tf_function, tf_util
  from tensorforce.core.policies import ParametrizedPolicy, ValuePolicy
label_desc: Use image node_id 7 for calling the ParametrizedValuePolicy obj's underlying member method code with example usage: obj.action_values() and returns: self, self, tf, self, tf, tf, tf, tf, tf