class_docstr: A neural-network based bandit agent for multi-objective optimization.
  
  This agent receives multiple neural networks. Each network will be trained by
  the agent to predict a specific objective. The agent also receives a
  Scalarizer, which transforms multiple predicted objectives to a single reward.
  The action is chosen greedily by the policy with respect to the scalarized
  predicted reward.
docstr: Creates a Greedy Multi-objective Neural Agent.
  
  Args:
    time_step_spec: A `TimeStep` spec of the expected time_steps.
    action_spec: A nest of `BoundedTensorSpec` representing the actions.
    scalarizer: A
      `tf_agents.bandits.multi_objective.multi_objective_scalarizer.Scalarizer`
      object that implements scalarization of multiple objectives into a
      single scalar reward.
    objective_network_and_loss_fn_sequence: A Sequence of Tuples
      (`tf_agents.network.Network`, error loss function) to be used by the
      agent. Each network `net` will be called as `net(observation,
      training=...)` and is expected to output a `tf.Tensor` of predicted
      values for a specific objective for all actions, shaped as [batch-size,
      number-of-actions]. Each network will be trained via minimizing the
      accompanying error loss function, which takes parameters labels,
      predictions, and weights (any function from tf.losses would work).
    optimizer: A 'tf.keras.optimizers.Optimizer' object, the optimizer to use
      for training.
    observation_and_action_constraint_splitter: A function used for masking
      valid/invalid actions with each state of the environment. The function
      takes in a full observation and returns a tuple consisting of 1) the
      part of the observation intended as input to the bandit agent and
      policy, and 2) the boolean mask of shape `[batch_size, num_actions]`.
      This function should also work with a `TensorSpec` as input, and should
      output `TensorSpec` objects for the observation and mask.
    accepts_per_arm_features: (bool) Whether the agent accepts per-arm
      features.
    gradient_clipping: A float representing the norm length to clip gradients
      (or None for no clipping.)
    debug_summaries: A Python bool, default False. When True, debug summaries
      are gathered.
    summarize_grads_and_vars: A Python bool, default False. When True,
      gradients and network variable summaries are written during training.
    enable_summaries: A Python bool, default True. When False, all summaries
      (debug or otherwise) should not be written.
    emit_policy_info: (tuple of strings) what side information we want to get
      as part of the policy info. Allowed values can be found in
      `policy_utilities.PolicyInfo`.
    train_step_counter: An optional `tf.Variable` to increment every time the
      train op is run.  Defaults to the `global_step`.
    laplacian_matrix: A float `Tensor` or a numpy array shaped `[num_actions,
      num_actions]`. This holds the Laplacian matrix used to regularize the
      smoothness of the estimated expected reward function. This only applies
      to problems where the actions have a graph structure. If `None`, the
      regularization is not applied.
    laplacian_smoothing_weights: A Sequence of floats that determines the
      per-objective weight of the regularization term. Note that this has no
      effect if `laplacian_matrix` above is `None`.
    name: Python str name of this agent. All variables in this module will
      fall under that name. Defaults to the class name.
  
  Raises:
    ValueError:
      - If the action spec contains more than one action or or it is not a
        bounded scalar int32 spec with minimum 0.
      - If the length of `objective_network_and_loss_fn_sequence` is less than
        two.
      - If the Laplacian matrix is provided and is invalid.
docstr_tok: ["Creates","a","Greedy","Multi-objective","Neural","Agent",".","Args",":","time_step_spec",":","A","`","TimeStep","`","spec","of","the","expected","time_steps",".","action_spec",":","A","nest","of","`","BoundedTensorSpec","`","representing","the","actions",".","scalarizer",":","A","`","tf_agents.bandits.multi_objective.multi_objective_scalarizer.Scalarizer","`","object","that","implements","scalarization","of","multiple","objectives","into","a","single","scalar","reward",".","objective_network_and_loss_fn_sequence",":","A","Sequence","of","Tuples","(","`","tf_agents.network.Network","`",",","error","loss","function",")","to","be","used","by","the","agent",".","Each","network","`","net","`","will","be","called","as","`","net","(","observation",",","training=","...",")","`","and","is","expected","to","output","a","`","tf.Tensor","`","of","predicted","values","for","a","specific","objective","for","all","actions",",","shaped","as","[","batch-size",",","number-of-actions","]",".","Each","network","will","be","trained","via","minimizing","the","accompanying","error","loss","function",",","which","takes","parameters","labels",",","predictions",",","and","weights","(","any","function","from","tf.losses","would","work",")",".","optimizer",":","A","'tf.keras.optimizers.Optimizer","'","object",",","the","optimizer","to","use","for","training",".","observation_and_action_constraint_splitter",":","A","function","used","for","masking","valid\/invalid","actions","with","each","state","of","the","environment",".","The","function","takes","in","a","full","observation","and","returns","a","tuple","consisting","of","1",")","the","part","of","the","observation","intended","as","input","to","the","bandit","agent","and","policy",",","and","2",")","the","boolean","mask","of","shape","`","[","batch_size",",","num_actions","]","`",".","This","function","should","also","work","with","a","`","TensorSpec","`","as","input",",","and","should","output","`","TensorSpec","`","objects","for","the","observation","and","mask",".","accepts_per_arm_features",":","(","bool",")","Whether","the","agent","accepts","per-arm","features",".","gradient_clipping",":","A","float","representing","the","norm","length","to","clip","gradients","(","or","None","for","no","clipping",".",")","debug_summaries",":","A","Python","bool",",","default","False",".","When","True",",","debug","summaries","are","gathered",".","summarize_grads_and_vars",":","A","Python","bool",",","default","False",".","When","True",",","gradients","and","network","variable","summaries","are","written","during","training",".","enable_summaries",":","A","Python","bool",",","default","True",".","When","False",",","all","summaries","(","debug","or","otherwise",")","should","not","be","written",".","emit_policy_info",":","(","tuple","of","strings",")","what","side","information","we","want","to","get","as","part","of","the","policy","info",".","Allowed","values","can","be","found","in","`","policy_utilities.PolicyInfo","`",".","train_step_counter",":","An","optional","`","tf.Variable","`","to","increment","every","time","the","train","op","is","run",".","Defaults","to","the","`","global_step","`",".","laplacian_matrix",":","A","float","`","Tensor","`","or","a","numpy","array","shaped","`","[","num_actions",",","num_actions","]","`",".","This","holds","the","Laplacian","matrix","used","to","regularize","the","smoothness","of","the","estimated","expected","reward","function",".","This","only","applies","to","problems","where","the","actions","have","a","graph","structure",".","If","`","None","`",",","the","regularization","is","not","applied",".","laplacian_smoothing_weights",":","A","Sequence","of","floats","that","determines","the","per-objective","weight","of","the","regularization","term",".","Note","that","this","has","no","effect","if","`","laplacian_matrix","`","above","is","`","None","`",".","name",":","Python","str","name","of","this","agent",".","All","variables","in","this","module","will","fall","under","that","name",".","Defaults","to","the","class","name",".","Raises",":","ValueError",":","-","If","the","action","spec","contains","more","than","one","action","or","or","it","is","not","a","bounded","scalar","int32","spec","with","minimum","0",".","-","If","the","length","of","`","objective_network_and_loss_fn_sequence","`","is","less","than","two",".","-","If","the","Laplacian","matrix","is","provided","and","is","invalid","."]
code: def __init__(
      self,
      time_step_spec: Optional[ts.TimeStep],
      action_spec: Optional[types.NestedBoundedTensorSpec],
      scalarizer: multi_objective_scalarizer.Scalarizer,
      objective_network_and_loss_fn_sequence: Sequence[
          Tuple[Network, Callable[..., tf.Tensor]]
      ],
      optimizer: tf.keras.optimizers.Optimizer,
      observation_and_action_constraint_splitter: types.Splitter = None,
      accepts_per_arm_features: bool = False,
      # Params for training.
      gradient_clipping: Optional[float] = None,
      # Params for debugging.
      debug_summaries: bool = False,
      summarize_grads_and_vars: bool = False,
      enable_summaries: bool = True,
      emit_policy_info: Tuple[Text, ...] = (),
      train_step_counter: Optional[tf.Variable] = None,
      laplacian_matrix: Optional[types.Float] = None,
      laplacian_smoothing_weights: Optional[Sequence[float]] = None,
      name: Optional[Text] = None,
  ):
      """Creates a Greedy Multi-objective Neural Agent.
  
      Args:
        time_step_spec: A `TimeStep` spec of the expected time_steps.
        action_spec: A nest of `BoundedTensorSpec` representing the actions.
        scalarizer: A
          `tf_agents.bandits.multi_objective.multi_objective_scalarizer.Scalarizer`
          object that implements scalarization of multiple objectives into a
          single scalar reward.
        objective_network_and_loss_fn_sequence: A Sequence of Tuples
          (`tf_agents.network.Network`, error loss function) to be used by the
          agent. Each network `net` will be called as `net(observation,
          training=...)` and is expected to output a `tf.Tensor` of predicted
          values for a specific objective for all actions, shaped as [batch-size,
          number-of-actions]. Each network will be trained via minimizing the
          accompanying error loss function, which takes parameters labels,
          predictions, and weights (any function from tf.losses would work).
        optimizer: A 'tf.keras.optimizers.Optimizer' object, the optimizer to use
          for training.
        observation_and_action_constraint_splitter: A function used for masking
          valid/invalid actions with each state of the environment. The function
          takes in a full observation and returns a tuple consisting of 1) the
          part of the observation intended as input to the bandit agent and
          policy, and 2) the boolean mask of shape `[batch_size, num_actions]`.
          This function should also work with a `TensorSpec` as input, and should
          output `TensorSpec` objects for the observation and mask.
        accepts_per_arm_features: (bool) Whether the agent accepts per-arm
          features.
        gradient_clipping: A float representing the norm length to clip gradients
          (or None for no clipping.)
        debug_summaries: A Python bool, default False. When True, debug summaries
          are gathered.
        summarize_grads_and_vars: A Python bool, default False. When True,
          gradients and network variable summaries are written during training.
        enable_summaries: A Python bool, default True. When False, all summaries
          (debug or otherwise) should not be written.
        emit_policy_info: (tuple of strings) what side information we want to get
          as part of the policy info. Allowed values can be found in
          `policy_utilities.PolicyInfo`.
        train_step_counter: An optional `tf.Variable` to increment every time the
          train op is run.  Defaults to the `global_step`.
        laplacian_matrix: A float `Tensor` or a numpy array shaped `[num_actions,
          num_actions]`. This holds the Laplacian matrix used to regularize the
          smoothness of the estimated expected reward function. This only applies
          to problems where the actions have a graph structure. If `None`, the
          regularization is not applied.
        laplacian_smoothing_weights: A Sequence of floats that determines the
          per-objective weight of the regularization term. Note that this has no
          effect if `laplacian_matrix` above is `None`.
        name: Python str name of this agent. All variables in this module will
          fall under that name. Defaults to the class name.
  
      Raises:
        ValueError:
          - If the action spec contains more than one action or or it is not a
            bounded scalar int32 spec with minimum 0.
          - If the length of `objective_network_and_loss_fn_sequence` is less than
            two.
          - If the Laplacian matrix is provided and is invalid.
      """
      tf.Module.__init__(self, name=name)
      common.tf_agents_gauge.get_cell("TFABandit").set(True)
      self._observation_and_action_constraint_splitter = (
          observation_and_action_constraint_splitter
      )
      self._num_actions = (
          policy_utilities.get_num_actions_from_tensor_spec(action_spec)
      )
      self._accepts_per_arm_features = accepts_per_arm_features
  
      self._num_objectives = len(objective_network_and_loss_fn_sequence)
      if self._num_objectives < 2:
          raise ValueError(
              "Number of objectives should be at least two, but found to be {}".format(
                  self._num_objectives
              )
          )
      self._objective_networks, self._error_loss_fns = tuple(
          zip(*objective_network_and_loss_fn_sequence)
      )
      self._optimizer = optimizer
      self._gradient_clipping = gradient_clipping
      self._heteroscedastic = [
          isinstance(
              network, heteroscedastic_q_network.HeteroscedasticQNetwork
          )
          for network in self._objective_networks
      ]
      self._laplacian_smoothing_weights = laplacian_smoothing_weights
      self._laplacian_matrix = None
      if laplacian_matrix is not None:
          self._laplacian_matrix = tf.convert_to_tensor(
              laplacian_matrix, dtype=tf.float32
          )
          # Check the validity of the laplacian matrix.
          tf.debugging.assert_near(
              0.0, tf.norm(tf.reduce_sum(self._laplacian_matrix, 1))
          )
          tf.debugging.assert_near(
              0.0, tf.norm(tf.reduce_sum(self._laplacian_matrix, 0))
          )
          tf.debugging.assert_near(
              0.0,
              tf.norm(
                  self._laplacian_matrix
                  - tf.linalg.matrix_transpose(self._laplacian_matrix)
              ),
          )
          if self._laplacian_smoothing_weights is None:
              raise ValueError(
                  "laplacian_smoothing_weights cannot be None when the "
                  "Laplacian matrix is provided."
              )
          if self._num_objectives != len(
              self._laplacian_smoothing_weights
          ):
              raise ValueError(
                  "The length of laplacian smoothing weights: {} "
                  "does not equal the number of objectives: {}".format(
                      len(self._laplacian_smoothing_weights),
                      self._num_objectives,
                  )
              )
  
      policy = greedy_multi_objective_policy.GreedyMultiObjectiveNeuralPolicy(
          time_step_spec,
          action_spec,
          scalarizer,
          self._objective_networks,
          observation_and_action_constraint_splitter,
          accepts_per_arm_features=accepts_per_arm_features,
          emit_policy_info=emit_policy_info,
      )
      training_data_spec = None
      if accepts_per_arm_features:
          training_data_spec = bandit_spec_utils.drop_arm_observation(
              policy.trajectory_spec
          )
  
      super(GreedyMultiObjectiveNeuralAgent, self).__init__(
          time_step_spec,
          action_spec,
          policy,
          collect_policy=policy,
          train_sequence_length=None,
          training_data_spec=training_data_spec,
          debug_summaries=debug_summaries,
          summarize_grads_and_vars=summarize_grads_and_vars,
          enable_summaries=enable_summaries,
          train_step_counter=train_step_counter,
      )
      self._as_trajectory = data_converter.AsTrajectory(
          self.data_context, sequence_length=None
      )
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  from typing import Callable, Optional, Sequence, Text, Tuple
  from absl import logging
  import gin
  import tensorflow
  from tf_agents.agents import data_converter
  from tf_agents.agents import tf_agent
  from tf_agents.bandits.agents import utils
  from tf_agents.bandits.multi_objective import multi_objective_scalarizer
  from tf_agents.bandits.networks import heteroscedastic_q_network
  from tf_agents.bandits.policies import greedy_multi_objective_neural_policy
  from tf_agents.bandits.specs import utils
  from tf_agents.networks.network import Network
  from tf_agents.policies import utils
  from tf_agents.trajectories import time_step
  from tf_agents.typing import types
  from tf_agents.utils import common
  from tf_agents.utils import eager_utils
label_desc: Use image node_id 1 to create a new GreedyMultiObjectiveNeuralAgent object from inherited base classes: tf_agent with example: obj = GreedyMultiObjectiveNeuralAgent(time_step_spec, action_spec, scalarizer, objective_network_and_loss_fn_sequence, optimizer, observation_and_action_constraint_splitter, accepts_per_arm_features, gradient_clipping, debug_summaries, summarize_grads_and_vars, enable_summaries, emit_policy_info, train_step_counter, laplacian_matrix, laplacian_smoothing_weights, name)