class_docstr: Policy that samples actions based on the FALCON algorithm.
class_docstr_tok: ["Policy","that","samples","actions","based","on","the","FALCON","algorithm","."]
docstr: Builds a FalconRewardPredictionPolicy given a reward network.
  
  This policy takes a tf_agents.Network predicting rewards and samples an
  action based on predicted rewards with the action distribution described
  in Step 6 of Algorithm 1 in the paper:
  
  David Simchi-Levi and Yunzong Xu, "Bypassing the Monster: A Faster and
  Simpler Optimal Algorithm for Contextual Bandits under Realizability",
  Mathematics of Operations Research, 2021.
  https://arxiv.org/pdf/2003.12699.pdf
  
  Args:
    time_step_spec: A `TimeStep` spec of the expected time_steps.
    action_spec: A nest of BoundedTensorSpec representing the actions.
    reward_network: An instance of a `tf_agents.network.Network`, callable via
      `network(observation, step_type) -> (output, final_state)`.
    exploitation_coefficient: float or callable that returns a float. Its
      value will be internally lower-bounded at 0. It controls how
      exploitative the policy behaves w.r.t the predicted rewards: A larger
      value makes the policy sample the greedy action (one with the best
      predicted reward) with a higher probability.
    max_exploration_probability_hint: An optional float, representing a hint
      on the maximum exploration probability, internally clipped to [0, 1].
      When this argument is set, `exploitation_coefficient` is ignored and the
      policy attempts to choose non-greedy actions with at most this
      probability. When such an upper bound cannot be achieved, e.g. due to
      insufficient training data, the policy attempts to minimize the
      probability of choosing non-greedy actions on a best-effort basis. For a
      demonstration of how it affects the policy behavior, see the unit test
      `testMaxExplorationProbabilityHint` in
      `falcon_reward_prediction_policy_test`.
    observation_and_action_constraint_splitter: A function used for masking
      valid/invalid actions with each state of the environment. The function
      takes in a full observation and returns a tuple consisting of 1) the
      part of the observation intended as input to the network and 2) the
      mask.  The mask should be a 0-1 `Tensor` of shape `[batch_size,
      num_actions]`. This function should also work with a `TensorSpec` as
      input, and should output `TensorSpec` objects for the observation and
      mask.
    accepts_per_arm_features: (bool) Whether the policy accepts per-arm
      features.
    constraints: iterable of constraints objects that are instances of
      `tf_agents.bandits.agents.BaseConstraint`.
    emit_policy_info: (tuple of strings) what side information we want to get
      as part of the policy info. Allowed values can be found in
      `policy_utilities.PolicyInfo`.
    num_samples_list: `Sequence` of tf.Variable's representing the number of
      examples for every action that the policy was trained with. For per-arm
      features, the size of the list is expected to be 1, representing the
      total number of examples the policy was trained with.
    name: The name of this policy. All variables in this module will fall
      under that name. Defaults to the class name.
  
  Raises:
    NotImplementedError: If `action_spec` contains more than one
      `BoundedTensorSpec` or the `BoundedTensorSpec` is not valid.
docstr_tok: ["Builds","a","FalconRewardPredictionPolicy","given","a","reward","network",".","This","policy","takes","a","tf_agents.Network","predicting","rewards","and","samples","an","action","based","on","predicted","rewards","with","the","action","distribution","described","in","Step","6","of","Algorithm","1","in","the","paper",":","David","Simchi-Levi","and","Yunzong","Xu",",","``","Bypassing","the","Monster",":","A","Faster","and","Simpler","Optimal","Algorithm","for","Contextual","Bandits","under","Realizability","''",",","Mathematics","of","Operations","Research",",","2021.","https",":","\/\/arxiv.org\/pdf\/2003.12699.pdf","Args",":","time_step_spec",":","A","`","TimeStep","`","spec","of","the","expected","time_steps",".","action_spec",":","A","nest","of","BoundedTensorSpec","representing","the","actions",".","reward_network",":","An","instance","of","a","`","tf_agents.network.Network","`",",","callable","via","`","network","(","observation",",","step_type",")","-",">","(","output",",","final_state",")","`",".","exploitation_coefficient",":","float","or","callable","that","returns","a","float",".","Its","value","will","be","internally","lower-bounded","at","0",".","It","controls","how","exploitative","the","policy","behaves","w.r.t","the","predicted","rewards",":","A","larger","value","makes","the","policy","sample","the","greedy","action","(","one","with","the","best","predicted","reward",")","with","a","higher","probability",".","max_exploration_probability_hint",":","An","optional","float",",","representing","a","hint","on","the","maximum","exploration","probability",",","internally","clipped","to","[","0",",","1","]",".","When","this","argument","is","set",",","`","exploitation_coefficient","`","is","ignored","and","the","policy","attempts","to","choose","non-greedy","actions","with","at","most","this","probability",".","When","such","an","upper","bound","can","not","be","achieved",",","e.g",".","due","to","insufficient","training","data",",","the","policy","attempts","to","minimize","the","probability","of","choosing","non-greedy","actions","on","a","best-effort","basis",".","For","a","demonstration","of","how","it","affects","the","policy","behavior",",","see","the","unit","test","`","testMaxExplorationProbabilityHint","`","in","`","falcon_reward_prediction_policy_test","`",".","observation_and_action_constraint_splitter",":","A","function","used","for","masking","valid\/invalid","actions","with","each","state","of","the","environment",".","The","function","takes","in","a","full","observation","and","returns","a","tuple","consisting","of","1",")","the","part","of","the","observation","intended","as","input","to","the","network","and","2",")","the","mask",".","The","mask","should","be","a","0-1","`","Tensor","`","of","shape","`","[","batch_size",",","num_actions","]","`",".","This","function","should","also","work","with","a","`","TensorSpec","`","as","input",",","and","should","output","`","TensorSpec","`","objects","for","the","observation","and","mask",".","accepts_per_arm_features",":","(","bool",")","Whether","the","policy","accepts","per-arm","features",".","constraints",":","iterable","of","constraints","objects","that","are","instances","of","`","tf_agents.bandits.agents.BaseConstraint","`",".","emit_policy_info",":","(","tuple","of","strings",")","what","side","information","we","want","to","get","as","part","of","the","policy","info",".","Allowed","values","can","be","found","in","`","policy_utilities.PolicyInfo","`",".","num_samples_list",":","`","Sequence","`","of","tf.Variable","'s","representing","the","number","of","examples","for","every","action","that","the","policy","was","trained","with",".","For","per-arm","features",",","the","size","of","the","list","is","expected","to","be","1",",","representing","the","total","number","of","examples","the","policy","was","trained","with",".","name",":","The","name","of","this","policy",".","All","variables","in","this","module","will","fall","under","that","name",".","Defaults","to","the","class","name",".","Raises",":","NotImplementedError",":","If","`","action_spec","`","contains","more","than","one","`","BoundedTensorSpec","`","or","the","`","BoundedTensorSpec","`","is","not","valid","."]
code: def __init__(
      self,
      time_step_spec: types.TimeStep,
      action_spec: types.NestedTensorSpec,
      reward_network: types.Network,
      exploitation_coefficient: Optional[
          types.FloatOrReturningFloat
      ] = 1.0,
      max_exploration_probability_hint: Optional[
          types.FloatOrReturningFloat
      ] = None,
      observation_and_action_constraint_splitter: Optional[
          types.Splitter
      ] = None,
      accepts_per_arm_features: bool = False,
      constraints: Iterable[constr.BaseConstraint] = (),
      emit_policy_info: Tuple[Text, ...] = (),
      num_samples_list: Sequence[tf.Variable] = (),
      name: Optional[Text] = None,
  ):
      """Builds a FalconRewardPredictionPolicy given a reward network.
  
      This policy takes a tf_agents.Network predicting rewards and samples an
      action based on predicted rewards with the action distribution described
      in Step 6 of Algorithm 1 in the paper:
  
      David Simchi-Levi and Yunzong Xu, "Bypassing the Monster: A Faster and
      Simpler Optimal Algorithm for Contextual Bandits under Realizability",
      Mathematics of Operations Research, 2021.
      https://arxiv.org/pdf/2003.12699.pdf
  
      Args:
        time_step_spec: A `TimeStep` spec of the expected time_steps.
        action_spec: A nest of BoundedTensorSpec representing the actions.
        reward_network: An instance of a `tf_agents.network.Network`, callable via
          `network(observation, step_type) -> (output, final_state)`.
        exploitation_coefficient: float or callable that returns a float. Its
          value will be internally lower-bounded at 0. It controls how
          exploitative the policy behaves w.r.t the predicted rewards: A larger
          value makes the policy sample the greedy action (one with the best
          predicted reward) with a higher probability.
        max_exploration_probability_hint: An optional float, representing a hint
          on the maximum exploration probability, internally clipped to [0, 1].
          When this argument is set, `exploitation_coefficient` is ignored and the
          policy attempts to choose non-greedy actions with at most this
          probability. When such an upper bound cannot be achieved, e.g. due to
          insufficient training data, the policy attempts to minimize the
          probability of choosing non-greedy actions on a best-effort basis. For a
          demonstration of how it affects the policy behavior, see the unit test
          `testMaxExplorationProbabilityHint` in
          `falcon_reward_prediction_policy_test`.
        observation_and_action_constraint_splitter: A function used for masking
          valid/invalid actions with each state of the environment. The function
          takes in a full observation and returns a tuple consisting of 1) the
          part of the observation intended as input to the network and 2) the
          mask.  The mask should be a 0-1 `Tensor` of shape `[batch_size,
          num_actions]`. This function should also work with a `TensorSpec` as
          input, and should output `TensorSpec` objects for the observation and
          mask.
        accepts_per_arm_features: (bool) Whether the policy accepts per-arm
          features.
        constraints: iterable of constraints objects that are instances of
          `tf_agents.bandits.agents.BaseConstraint`.
        emit_policy_info: (tuple of strings) what side information we want to get
          as part of the policy info. Allowed values can be found in
          `policy_utilities.PolicyInfo`.
        num_samples_list: `Sequence` of tf.Variable's representing the number of
          examples for every action that the policy was trained with. For per-arm
          features, the size of the list is expected to be 1, representing the
          total number of examples the policy was trained with.
        name: The name of this policy. All variables in this module will fall
          under that name. Defaults to the class name.
  
      Raises:
        NotImplementedError: If `action_spec` contains more than one
          `BoundedTensorSpec` or the `BoundedTensorSpec` is not valid.
      """
      super(FalconRewardPredictionPolicy, self).__init__(
          time_step_spec,
          action_spec,
          reward_network,
          observation_and_action_constraint_splitter,
          accepts_per_arm_features,
          constraints,
          emit_policy_info,
          name,
      )
  
      self._exploitation_coefficient = exploitation_coefficient
      self._max_exploration_probability_hint = (
          max_exploration_probability_hint
      )
      if num_samples_list:
          self._num_samples_list = num_samples_list
      else:
          self._num_samples_list = [tf.Variable(0, dtype=tf.int64)] * (
              1
              if self.accepts_per_arm_features
              else self._expected_num_actions
          )
      if (
          self.accepts_per_arm_features
          and len(self._num_samples_list) != 1
      ):
          raise ValueError(
              "num_samples_list is expected to be of length 1 when"
              "accepts_per_arm_features is True, but found otherwise: "
              f"{self._num_samples_list} "
          )
      if not self.accepts_per_arm_features and (
          len(self._num_samples_list) != self._expected_num_actions
      ):
          raise ValueError(
              "Size of num_samples_list: ",
              len(self._num_samples_list),
              " does not match the expected number of actions:",
              self._expected_num_actions,
          )
      self._num_trainable_elements = get_number_of_trainable_elements(
          self._reward_network
      )
  
file_path: agents/tf_agents/bandits/policies/falcon_reward_prediction_policy.py
imports: from typing import Iterable, Optional, Sequence, Text, Tuple
  import tensorflow
  import tensorflow_probability
  from tf_agents.bandits.policies import constraints
  from tf_agents.bandits.policies import reward_prediction_base_policy
  from tf_agents.distributions import shifted_categorical
  from tf_agents.policies import utils
  from tf_agents.typing import types
label_desc: Use image node_id 1 to create a new FalconRewardPredictionPolicy object from inherited base classes: reward_prediction_base_policy with example: obj = FalconRewardPredictionPolicy(time_step_spec, action_spec, reward_network, exploitation_coefficient, max_exploration_probability_hint, observation_and_action_constraint_splitter, accepts_per_arm_features, constraints, emit_policy_info, num_samples_list, name)