class_docstr: A neural reward network based bandit agent.
  
  This agent receives a neural network that it trains to predict rewards. The
  action is chosen greedily with respect to the prediction.
class_docstr_tok: ["A","neural","reward","network","based","bandit","agent",".","This","agent","receives","a","neural","network","that","it","trains","to","predict","rewards",".","The","action","is","chosen","greedily","with","respect","to","the","prediction","."]
docstr: Creates a Greedy Reward Network Prediction Agent.
  
   In some use cases, the actions are not independent and they are related to
   each other (e.g., when the actions are ordinal integers). Assuming that
   the relations between arms can be modeled by a graph, we may want to
   enforce that the estimated reward function is smooth over the graph. This
   implies that the estimated rewards `r_i` and `r_j` for two related actions
   `i` and `j`, should be close to each other. To quantify this smoothness
   criterion we use the Laplacian matrix `L` of the graph over the actions.
   When the laplacian smoothing is enabled, the loss is extended to:
   ```
     Loss_new := Loss + lambda r^T * L * r,
   ```
   where `r` is the estimated reward vector for all actions. The second
   term is the laplacian smoothing regularization term and `lambda` is the
   weight that determines how strongly we enforce the regularization.
   For more details, please see:
   "Bandits on graphs and structures", Michal Valko
   https://hal.inria.fr/tel-01359757/document
  
  Args:
    time_step_spec: A `TimeStep` spec of the expected time_steps.
    action_spec: A nest of `BoundedTensorSpec` representing the actions.
    reward_network: A `tf_agents.network.Network` to be used by the agent. The
      network will be called with call(observation, step_type) and it is
      expected to provide a reward prediction for all actions.
    optimizer: The optimizer to use for training.
    observation_and_action_constraint_splitter: A function used for masking
      valid/invalid actions with each state of the environment. The function
      takes in a full observation and returns a tuple consisting of 1) the
      part of the observation intended as input to the bandit agent and
      policy, and 2) the boolean mask. This function should also work with a
      `TensorSpec` as input, and should output `TensorSpec` objects for the
      observation and mask.
    accepts_per_arm_features: (bool) Whether the policy accepts per-arm
      features.
    constraints: iterable of constraints objects that are instances of
      `tf_agents.bandits.agents.NeuralConstraint`.
    error_loss_fn: A function for computing the error loss, taking parameters
      labels, predictions, and weights (any function from tf.losses would
      work). The default is `tf.losses.mean_squared_error`.
    gradient_clipping: A float representing the norm length to clip gradients
      (or None for no clipping.)
    debug_summaries: A Python bool, default False. When True, debug summaries
      are gathered.
    summarize_grads_and_vars: A Python bool, default False. When True,
      gradients and network variable summaries are written during training.
    enable_summaries: A Python bool, default True. When False, all summaries
      (debug or otherwise) should not be written.
    emit_policy_info: (tuple of strings) what side information we want to get
      as part of the policy info. Allowed values can be found in
      `policy_utilities.PolicyInfo`.
    train_step_counter: An optional `tf.Variable` to increment every time the
      train op is run.  Defaults to the `global_step`.
    num_samples_list: An optional list or tuple of tf.Variable's. It holds the
      number of samples per action. If provided, it will be populated. For
      per-arm features, it is expected to have only one element, which holds
      the total number of samples.
    laplacian_matrix: A float `Tensor` or a numpy array shaped `[num_actions,
      num_actions]`. This holds the Laplacian matrix used to regularize the
      smoothness of the estimated expected reward function. This only applies
      to problems where the actions have a graph structure. If `None`, the
      regularization is not applied.
    laplacian_smoothing_weight: A float that determines the weight of the
      regularization term. Note that this has no effect if `laplacian_matrix`
      above is `None`.
    name: Python str name of this agent. All variables in this module will
      fall under that name. Defaults to the class name.
  
  Raises:
    ValueError: If the action spec contains more than one action or or it is
    not a bounded scalar int32 spec with minimum 0.
    InvalidArgumentError: if the Laplacian provided is not None and not valid.
docstr_tok: ["Creates","a","Greedy","Reward","Network","Prediction","Agent",".","In","some","use","cases",",","the","actions","are","not","independent","and","they","are","related","to","each","other","(","e.g.",",","when","the","actions","are","ordinal","integers",")",".","Assuming","that","the","relations","between","arms","can","be","modeled","by","a","graph",",","we","may","want","to","enforce","that","the","estimated","reward","function","is","smooth","over","the","graph",".","This","implies","that","the","estimated","rewards","`","r_i","`","and","`","r_j","`","for","two","related","actions","`","i","`","and","`","j","`",",","should","be","close","to","each","other",".","To","quantify","this","smoothness","criterion","we","use","the","Laplacian","matrix","`","L","`","of","the","graph","over","the","actions",".","When","the","laplacian","smoothing","is","enabled",",","the","loss","is","extended","to",":","``","`","Loss_new",":","=","Loss","+","lambda","r^T","*","L","*","r",",","``","`","where","`","r","`","is","the","estimated","reward","vector","for","all","actions",".","The","second","term","is","the","laplacian","smoothing","regularization","term","and","`","lambda","`","is","the","weight","that","determines","how","strongly","we","enforce","the","regularization",".","For","more","details",",","please","see",":","``","Bandits","on","graphs","and","structures","''",",","Michal","Valko","https",":","\/\/hal.inria.fr\/tel-01359757\/document","Args",":","time_step_spec",":","A","`","TimeStep","`","spec","of","the","expected","time_steps",".","action_spec",":","A","nest","of","`","BoundedTensorSpec","`","representing","the","actions",".","reward_network",":","A","`","tf_agents.network.Network","`","to","be","used","by","the","agent",".","The","network","will","be","called","with","call","(","observation",",","step_type",")","and","it","is","expected","to","provide","a","reward","prediction","for","all","actions",".","optimizer",":","The","optimizer","to","use","for","training",".","observation_and_action_constraint_splitter",":","A","function","used","for","masking","valid\/invalid","actions","with","each","state","of","the","environment",".","The","function","takes","in","a","full","observation","and","returns","a","tuple","consisting","of","1",")","the","part","of","the","observation","intended","as","input","to","the","bandit","agent","and","policy",",","and","2",")","the","boolean","mask",".","This","function","should","also","work","with","a","`","TensorSpec","`","as","input",",","and","should","output","`","TensorSpec","`","objects","for","the","observation","and","mask",".","accepts_per_arm_features",":","(","bool",")","Whether","the","policy","accepts","per-arm","features",".","constraints",":","iterable","of","constraints","objects","that","are","instances","of","`","tf_agents.bandits.agents.NeuralConstraint","`",".","error_loss_fn",":","A","function","for","computing","the","error","loss",",","taking","parameters","labels",",","predictions",",","and","weights","(","any","function","from","tf.losses","would","work",")",".","The","default","is","`","tf.losses.mean_squared_error","`",".","gradient_clipping",":","A","float","representing","the","norm","length","to","clip","gradients","(","or","None","for","no","clipping",".",")","debug_summaries",":","A","Python","bool",",","default","False",".","When","True",",","debug","summaries","are","gathered",".","summarize_grads_and_vars",":","A","Python","bool",",","default","False",".","When","True",",","gradients","and","network","variable","summaries","are","written","during","training",".","enable_summaries",":","A","Python","bool",",","default","True",".","When","False",",","all","summaries","(","debug","or","otherwise",")","should","not","be","written",".","emit_policy_info",":","(","tuple","of","strings",")","what","side","information","we","want","to","get","as","part","of","the","policy","info",".","Allowed","values","can","be","found","in","`","policy_utilities.PolicyInfo","`",".","train_step_counter",":","An","optional","`","tf.Variable","`","to","increment","every","time","the","train","op","is","run",".","Defaults","to","the","`","global_step","`",".","num_samples_list",":","An","optional","list","or","tuple","of","tf.Variable","'s",".","It","holds","the","number","of","samples","per","action",".","If","provided",",","it","will","be","populated",".","For","per-arm","features",",","it","is","expected","to","have","only","one","element",",","which","holds","the","total","number","of","samples",".","laplacian_matrix",":","A","float","`","Tensor","`","or","a","numpy","array","shaped","`","[","num_actions",",","num_actions","]","`",".","This","holds","the","Laplacian","matrix","used","to","regularize","the","smoothness","of","the","estimated","expected","reward","function",".","This","only","applies","to","problems","where","the","actions","have","a","graph","structure",".","If","`","None","`",",","the","regularization","is","not","applied",".","laplacian_smoothing_weight",":","A","float","that","determines","the","weight","of","the","regularization","term",".","Note","that","this","has","no","effect","if","`","laplacian_matrix","`","above","is","`","None","`",".","name",":","Python","str","name","of","this","agent",".","All","variables","in","this","module","will","fall","under","that","name",".","Defaults","to","the","class","name",".","Raises",":","ValueError",":","If","the","action","spec","contains","more","than","one","action","or","or","it","is","not","a","bounded","scalar","int32","spec","with","minimum","0",".","InvalidArgumentError",":","if","the","Laplacian","provided","is","not","None","and","not","valid","."]
code: def __init__(
      self,
      time_step_spec: types.TimeStep,
      action_spec: types.BoundedTensorSpec,
      reward_network: types.Network,
      optimizer: types.Optimizer,
      observation_and_action_constraint_splitter: Optional[
          types.Splitter
      ] = None,
      accepts_per_arm_features: bool = False,
      constraints: Iterable[constr.BaseConstraint] = (),
      # Params for training.
      error_loss_fn: types.LossFn = tf.compat.v1.losses.mean_squared_error,
      gradient_clipping: Optional[float] = None,
      # Params for debugging.
      debug_summaries: bool = False,
      summarize_grads_and_vars: bool = False,
      enable_summaries: bool = True,
      emit_policy_info: Tuple[Text, ...] = (),
      train_step_counter: Optional[tf.Variable] = None,
      num_samples_list: Sequence[tf.Variable] = (),
      laplacian_matrix: Optional[types.Float] = None,
      laplacian_smoothing_weight: float = 0.001,
      name: Optional[Text] = None,
  ):
      """Creates a Greedy Reward Network Prediction Agent.
  
       In some use cases, the actions are not independent and they are related to
       each other (e.g., when the actions are ordinal integers). Assuming that
       the relations between arms can be modeled by a graph, we may want to
       enforce that the estimated reward function is smooth over the graph. This
       implies that the estimated rewards `r_i` and `r_j` for two related actions
       `i` and `j`, should be close to each other. To quantify this smoothness
       criterion we use the Laplacian matrix `L` of the graph over the actions.
       When the laplacian smoothing is enabled, the loss is extended to:
       ```
         Loss_new := Loss + lambda r^T * L * r,
       ```
       where `r` is the estimated reward vector for all actions. The second
       term is the laplacian smoothing regularization term and `lambda` is the
       weight that determines how strongly we enforce the regularization.
       For more details, please see:
       "Bandits on graphs and structures", Michal Valko
       https://hal.inria.fr/tel-01359757/document
  
      Args:
        time_step_spec: A `TimeStep` spec of the expected time_steps.
        action_spec: A nest of `BoundedTensorSpec` representing the actions.
        reward_network: A `tf_agents.network.Network` to be used by the agent. The
          network will be called with call(observation, step_type) and it is
          expected to provide a reward prediction for all actions.
        optimizer: The optimizer to use for training.
        observation_and_action_constraint_splitter: A function used for masking
          valid/invalid actions with each state of the environment. The function
          takes in a full observation and returns a tuple consisting of 1) the
          part of the observation intended as input to the bandit agent and
          policy, and 2) the boolean mask. This function should also work with a
          `TensorSpec` as input, and should output `TensorSpec` objects for the
          observation and mask.
        accepts_per_arm_features: (bool) Whether the policy accepts per-arm
          features.
        constraints: iterable of constraints objects that are instances of
          `tf_agents.bandits.agents.NeuralConstraint`.
        error_loss_fn: A function for computing the error loss, taking parameters
          labels, predictions, and weights (any function from tf.losses would
          work). The default is `tf.losses.mean_squared_error`.
        gradient_clipping: A float representing the norm length to clip gradients
          (or None for no clipping.)
        debug_summaries: A Python bool, default False. When True, debug summaries
          are gathered.
        summarize_grads_and_vars: A Python bool, default False. When True,
          gradients and network variable summaries are written during training.
        enable_summaries: A Python bool, default True. When False, all summaries
          (debug or otherwise) should not be written.
        emit_policy_info: (tuple of strings) what side information we want to get
          as part of the policy info. Allowed values can be found in
          `policy_utilities.PolicyInfo`.
        train_step_counter: An optional `tf.Variable` to increment every time the
          train op is run.  Defaults to the `global_step`.
        num_samples_list: An optional list or tuple of tf.Variable's. It holds the
          number of samples per action. If provided, it will be populated. For
          per-arm features, it is expected to have only one element, which holds
          the total number of samples.
        laplacian_matrix: A float `Tensor` or a numpy array shaped `[num_actions,
          num_actions]`. This holds the Laplacian matrix used to regularize the
          smoothness of the estimated expected reward function. This only applies
          to problems where the actions have a graph structure. If `None`, the
          regularization is not applied.
        laplacian_smoothing_weight: A float that determines the weight of the
          regularization term. Note that this has no effect if `laplacian_matrix`
          above is `None`.
        name: Python str name of this agent. All variables in this module will
          fall under that name. Defaults to the class name.
  
      Raises:
        ValueError: If the action spec contains more than one action or or it is
        not a bounded scalar int32 spec with minimum 0.
        InvalidArgumentError: if the Laplacian provided is not None and not valid.
      """
      tf.Module.__init__(self, name=name)
      common.tf_agents_gauge.get_cell("TFABandit").set(True)
      self._observation_and_action_constraint_splitter = (
          observation_and_action_constraint_splitter
      )
      self._num_actions = (
          policy_utilities.get_num_actions_from_tensor_spec(action_spec)
      )
      self._accepts_per_arm_features = accepts_per_arm_features
      self._constraints = constraints
      if num_samples_list:
          if accepts_per_arm_features and (len(num_samples_list) != 1):
              raise ValueError(
                  "num_samples_list is expected to be of length 1 when ",
                  "accepts_per_arm_features is True, but is found "
                  f"otherwise: {num_samples_list}",
              )
          if (not accepts_per_arm_features) and (
              len(num_samples_list) != self._num_actions
          ):
              raise ValueError(
                  "num_samples_list is expected to have length equal to the ",
                  "number of actions: ",
                  self._num_actions,
                  " , but found to be",
                  len(num_samples_list),
              )
  
      self._num_samples_list = num_samples_list
  
      reward_network.create_variables()
      self._reward_network = reward_network
      self._optimizer = optimizer
      self._error_loss_fn = error_loss_fn
      self._gradient_clipping = gradient_clipping
      self._heteroscedastic = isinstance(
          reward_network,
          heteroscedastic_q_network.HeteroscedasticQNetwork,
      )
      self._laplacian_matrix = None
      if laplacian_matrix is not None:
          self._laplacian_matrix = tf.convert_to_tensor(
              laplacian_matrix, dtype=tf.float32
          )
          # Check the validity of the laplacian matrix.
          tf.debugging.assert_near(
              0.0, tf.norm(tf.reduce_sum(self._laplacian_matrix, 1))
          )
          tf.debugging.assert_near(
              0.0, tf.norm(tf.reduce_sum(self._laplacian_matrix, 0))
          )
      self._laplacian_smoothing_weight = laplacian_smoothing_weight
  
      policy = greedy_reward_policy.GreedyRewardPredictionPolicy(
          time_step_spec,
          action_spec,
          reward_network,
          observation_and_action_constraint_splitter,
          constraints=constraints,
          accepts_per_arm_features=accepts_per_arm_features,
          emit_policy_info=emit_policy_info,
      )
      training_data_spec = None
      if accepts_per_arm_features:
          training_data_spec = bandit_spec_utils.drop_arm_observation(
              policy.trajectory_spec
          )
  
      super(GreedyRewardPredictionAgent, self).__init__(
          time_step_spec,
          action_spec,
          policy,
          collect_policy=policy,
          train_sequence_length=None,
          training_data_spec=training_data_spec,
          debug_summaries=debug_summaries,
          summarize_grads_and_vars=summarize_grads_and_vars,
          enable_summaries=enable_summaries,
          train_step_counter=train_step_counter,
      )
      self._as_trajectory = data_converter.AsTrajectory(
          self.data_context, sequence_length=None
      )
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  from typing import Iterable, Optional, Sequence, Text, Tuple
  from absl import logging
  import gin
  import tensorflow
  from tf_agents.agents import data_converter
  from tf_agents.agents import tf_agent
  from tf_agents.bandits.agents import utils
  from tf_agents.bandits.networks import heteroscedastic_q_network
  from tf_agents.bandits.policies import constraints
  from tf_agents.bandits.policies import greedy_reward_prediction_policy
  from tf_agents.bandits.specs import utils
  from tf_agents.policies import utils
  from tf_agents.typing import types
  from tf_agents.utils import common
  from tf_agents.utils import eager_utils
label_desc: Use image node_id 1 to create a new GreedyRewardPredictionAgent object from inherited base classes: tf_agent with example: obj = GreedyRewardPredictionAgent(time_step_spec, action_spec, reward_network, optimizer, observation_and_action_constraint_splitter, accepts_per_arm_features, constraints, error_loss_fn, gradient_clipping, debug_summaries, summarize_grads_and_vars, enable_summaries, emit_policy_info, train_step_counter, num_samples_list, laplacian_matrix, laplacian_smoothing_weight, name)