class_docstr: Linear Bandit Policy to be used by LinUCB, LinTS and possibly others.
class_docstr_tok: ["Linear","Bandit","Policy","to","be","used","by","LinUCB",",","LinTS","and","possibly","others","."]
docstr: Initializes `LinearBanditPolicy`.
  
  The `cov_matrix`, `data_vector`, `num_samples`, and `theta` arguments may be
  either `Tensor`s or `tf.Variable`s. If they are variables, any assignments
  to those variables will be reflected in the output of the policy.
  
  Args:
    action_spec: `TensorSpec` containing action specification.
    cov_matrix: list of the covariance matrices A in the paper. If the policy
      accepts per-arm features, the length of this list is 1, as there is only
      one model. Otherwise, there is one A matrix per arm.
    data_vector: list of the b vectors in the paper. The b vector is a
      weighted sum of the observations, where the weight is the corresponding
      reward. If the policy accepts per-arm features, this list should be of
      length 1, as there only 1 reward model maintained. Otherwise, each arm
      has its own vector b.
    num_samples: list of number of samples per arm, unless the policy accepts
      per-arm features, in which case this is just the number of samples seen.
    time_step_spec: A `TimeStep` spec of the expected time_steps.
    exploration_strategy: An Enum of type ExplortionStrategy. The strategy
      used for choosing the actions to incorporate exploration. Currently
      supported strategies are `optimistic` and `sampling`.
    alpha: a float value used to scale the confidence intervals.
    eig_vals: list of eigenvalues for each covariance matrix (one per arm,
      unless the policy accepts per-arm features).
    eig_matrix: list of eigenvectors for each covariance matrix (one per arm,
      unless the policy accepts per-arm features).
    tikhonov_weight: (float) tikhonov regularization term.
    add_bias: If true, a bias term will be added to the linear reward
      estimation.
    emit_policy_info: (tuple of strings) what side information we want to get
      as part of the policy info. Allowed values can be found in
      `policy_utilities.PolicyInfo`.
    emit_log_probability: Whether to emit log probabilities.
    accepts_per_arm_features: (bool) Whether the policy accepts per-arm
      features.
    observation_and_action_constraint_splitter: A function used for masking
      valid/invalid actions with each state of the environment. The function
      takes in a full observation and returns a tuple consisting of 1) the
      part of the observation intended as input to the bandit policy and 2)
      the mask. The mask should be a 0-1 `Tensor` of shape `[batch_size,
      num_actions]`. This function should also work with a `TensorSpec` as
      input, and should output `TensorSpec` objects for the observation and
      mask.
    theta: An optional 2-d tf.Tensor of the theta vectors shaped as `[k, n]`,
      where k denotes the number of arms and n denotes the overall context
      dimension. When `accepts_per_arm_features` is true, k is expected to be
      1 and n is the total dimension of the (flattened) global features and
      the (flattened) per-arm features. When supplied, the policy assumes it's
      consistent with the value computed from the other arguments
      `cov_matrix`, `data_vector`, and `tikhonov_weight`. If that is not the
      case, the policy may behave unexpectedly. Supplying pre-computed theta
      is the most useful for users who desire a greedy policy that selects
      actions solely based on the theta vectors, because this may
      significantly reduce the policy's inference latency.
    name: The name of this policy.
docstr_tok: ["Initializes","`","LinearBanditPolicy","`",".","The","`","cov_matrix","`",",","`","data_vector","`",",","`","num_samples","`",",","and","`","theta","`","arguments","may","be","either","`","Tensor","`","s","or","`","tf.Variable","`","s",".","If","they","are","variables",",","any","assignments","to","those","variables","will","be","reflected","in","the","output","of","the","policy",".","Args",":","action_spec",":","`","TensorSpec","`","containing","action","specification",".","cov_matrix",":","list","of","the","covariance","matrices","A","in","the","paper",".","If","the","policy","accepts","per-arm","features",",","the","length","of","this","list","is","1",",","as","there","is","only","one","model",".","Otherwise",",","there","is","one","A","matrix","per","arm",".","data_vector",":","list","of","the","b","vectors","in","the","paper",".","The","b","vector","is","a","weighted","sum","of","the","observations",",","where","the","weight","is","the","corresponding","reward",".","If","the","policy","accepts","per-arm","features",",","this","list","should","be","of","length","1",",","as","there","only","1","reward","model","maintained",".","Otherwise",",","each","arm","has","its","own","vector","b.","num_samples",":","list","of","number","of","samples","per","arm",",","unless","the","policy","accepts","per-arm","features",",","in","which","case","this","is","just","the","number","of","samples","seen",".","time_step_spec",":","A","`","TimeStep","`","spec","of","the","expected","time_steps",".","exploration_strategy",":","An","Enum","of","type","ExplortionStrategy",".","The","strategy","used","for","choosing","the","actions","to","incorporate","exploration",".","Currently","supported","strategies","are","`","optimistic","`","and","`","sampling","`",".","alpha",":","a","float","value","used","to","scale","the","confidence","intervals",".","eig_vals",":","list","of","eigenvalues","for","each","covariance","matrix","(","one","per","arm",",","unless","the","policy","accepts","per-arm","features",")",".","eig_matrix",":","list","of","eigenvectors","for","each","covariance","matrix","(","one","per","arm",",","unless","the","policy","accepts","per-arm","features",")",".","tikhonov_weight",":","(","float",")","tikhonov","regularization","term",".","add_bias",":","If","true",",","a","bias","term","will","be","added","to","the","linear","reward","estimation",".","emit_policy_info",":","(","tuple","of","strings",")","what","side","information","we","want","to","get","as","part","of","the","policy","info",".","Allowed","values","can","be","found","in","`","policy_utilities.PolicyInfo","`",".","emit_log_probability",":","Whether","to","emit","log","probabilities",".","accepts_per_arm_features",":","(","bool",")","Whether","the","policy","accepts","per-arm","features",".","observation_and_action_constraint_splitter",":","A","function","used","for","masking","valid\/invalid","actions","with","each","state","of","the","environment",".","The","function","takes","in","a","full","observation","and","returns","a","tuple","consisting","of","1",")","the","part","of","the","observation","intended","as","input","to","the","bandit","policy","and","2",")","the","mask",".","The","mask","should","be","a","0-1","`","Tensor","`","of","shape","`","[","batch_size",",","num_actions","]","`",".","This","function","should","also","work","with","a","`","TensorSpec","`","as","input",",","and","should","output","`","TensorSpec","`","objects","for","the","observation","and","mask",".","theta",":","An","optional","2-d","tf.Tensor","of","the","theta","vectors","shaped","as","`","[","k",",","n","]","`",",","where","k","denotes","the","number","of","arms","and","n","denotes","the","overall","context","dimension",".","When","`","accepts_per_arm_features","`","is","true",",","k","is","expected","to","be","1","and","n","is","the","total","dimension","of","the","(","flattened",")","global","features","and","the","(","flattened",")","per-arm","features",".","When","supplied",",","the","policy","assumes","it's","consistent","with","the","value","computed","from","the","other","arguments","`","cov_matrix","`",",","`","data_vector","`",",","and","`","tikhonov_weight","`",".","If","that","is","not","the","case",",","the","policy","may","behave","unexpectedly",".","Supplying","pre-computed","theta","is","the","most","useful","for","users","who","desire","a","greedy","policy","that","selects","actions","solely","based","on","the","theta","vectors",",","because","this","may","significantly","reduce","the","policy","'s","inference","latency",".","name",":","The","name","of","this","policy","."]
code: def __init__(
      self,
      action_spec: types.BoundedTensorSpec,
      cov_matrix: Sequence[types.Float],
      data_vector: Sequence[types.Float],
      num_samples: Sequence[types.Int],
      time_step_spec: Optional[types.TimeStep] = None,
      exploration_strategy: ExplorationStrategy = ExplorationStrategy.optimistic,
      alpha: float = 1.0,
      eig_vals: Sequence[types.Float] = (),
      eig_matrix: Sequence[types.Float] = (),
      tikhonov_weight: float = 1.0,
      add_bias: bool = False,
      emit_policy_info: Sequence[Text] = (),
      emit_log_probability: bool = False,
      accepts_per_arm_features: bool = False,
      observation_and_action_constraint_splitter: Optional[
          types.Splitter
      ] = None,
      theta: Optional[types.Tensor] = None,
      name: Optional[Text] = None,
  ):
      """Initializes `LinearBanditPolicy`.
  
      The `cov_matrix`, `data_vector`, `num_samples`, and `theta` arguments may be
      either `Tensor`s or `tf.Variable`s. If they are variables, any assignments
      to those variables will be reflected in the output of the policy.
  
      Args:
        action_spec: `TensorSpec` containing action specification.
        cov_matrix: list of the covariance matrices A in the paper. If the policy
          accepts per-arm features, the length of this list is 1, as there is only
          one model. Otherwise, there is one A matrix per arm.
        data_vector: list of the b vectors in the paper. The b vector is a
          weighted sum of the observations, where the weight is the corresponding
          reward. If the policy accepts per-arm features, this list should be of
          length 1, as there only 1 reward model maintained. Otherwise, each arm
          has its own vector b.
        num_samples: list of number of samples per arm, unless the policy accepts
          per-arm features, in which case this is just the number of samples seen.
        time_step_spec: A `TimeStep` spec of the expected time_steps.
        exploration_strategy: An Enum of type ExplortionStrategy. The strategy
          used for choosing the actions to incorporate exploration. Currently
          supported strategies are `optimistic` and `sampling`.
        alpha: a float value used to scale the confidence intervals.
        eig_vals: list of eigenvalues for each covariance matrix (one per arm,
          unless the policy accepts per-arm features).
        eig_matrix: list of eigenvectors for each covariance matrix (one per arm,
          unless the policy accepts per-arm features).
        tikhonov_weight: (float) tikhonov regularization term.
        add_bias: If true, a bias term will be added to the linear reward
          estimation.
        emit_policy_info: (tuple of strings) what side information we want to get
          as part of the policy info. Allowed values can be found in
          `policy_utilities.PolicyInfo`.
        emit_log_probability: Whether to emit log probabilities.
        accepts_per_arm_features: (bool) Whether the policy accepts per-arm
          features.
        observation_and_action_constraint_splitter: A function used for masking
          valid/invalid actions with each state of the environment. The function
          takes in a full observation and returns a tuple consisting of 1) the
          part of the observation intended as input to the bandit policy and 2)
          the mask. The mask should be a 0-1 `Tensor` of shape `[batch_size,
          num_actions]`. This function should also work with a `TensorSpec` as
          input, and should output `TensorSpec` objects for the observation and
          mask.
        theta: An optional 2-d tf.Tensor of the theta vectors shaped as `[k, n]`,
          where k denotes the number of arms and n denotes the overall context
          dimension. When `accepts_per_arm_features` is true, k is expected to be
          1 and n is the total dimension of the (flattened) global features and
          the (flattened) per-arm features. When supplied, the policy assumes it's
          consistent with the value computed from the other arguments
          `cov_matrix`, `data_vector`, and `tikhonov_weight`. If that is not the
          case, the policy may behave unexpectedly. Supplying pre-computed theta
          is the most useful for users who desire a greedy policy that selects
          actions solely based on the theta vectors, because this may
          significantly reduce the policy's inference latency.
        name: The name of this policy.
      """
      policy_utilities.check_no_mask_with_arm_features(
          accepts_per_arm_features,
          observation_and_action_constraint_splitter,
      )
      if not isinstance(cov_matrix, (list, tuple)):
          raise ValueError(
              "cov_matrix must be a list of matrices (Tensors)."
          )
      self._cov_matrix = cov_matrix
  
      if not isinstance(data_vector, (list, tuple)):
          raise ValueError(
              "data_vector must be a list of vectors (Tensors)."
          )
      self._data_vector = data_vector
  
      if not isinstance(num_samples, (list, tuple)):
          raise ValueError(
              "num_samples must be a list of vectors (Tensors)."
          )
      self._num_samples = num_samples
  
      if not isinstance(eig_vals, (list, tuple)):
          raise ValueError(
              "eig_vals must be a list of vectors (Tensors)."
          )
      self._eig_vals = eig_vals
  
      if not isinstance(eig_matrix, (list, tuple)):
          raise ValueError(
              "eig_matrix must be a list of vectors (Tensors)."
          )
      self._eig_matrix = eig_matrix
  
      self._exploration_strategy = exploration_strategy
      if exploration_strategy == ExplorationStrategy.sampling:
          # We do not have a way to calculate log probabilities for TS yet.
          emit_log_probability = False
  
      self._alpha = alpha
      self._use_eigendecomp = False
      if eig_matrix:
          self._use_eigendecomp = True
      self._tikhonov_weight = tikhonov_weight
      self._add_bias = add_bias
      self._accepts_per_arm_features = accepts_per_arm_features
      if tf.nest.is_nested(action_spec):
          raise ValueError("Nested `action_spec` is not supported.")
      self._num_actions = (
          policy_utilities.get_num_actions_from_tensor_spec(action_spec)
      )
      self._check_input_variables()
      if observation_and_action_constraint_splitter is not None:
          context_spec, _ = observation_and_action_constraint_splitter(
              time_step_spec.observation
          )
      else:
          context_spec = time_step_spec.observation
      (
          self._global_context_dim,
          self._arm_context_dim,
      ) = bandit_spec_utils.get_context_dims_from_spec(
          context_spec, accepts_per_arm_features
      )
  
      if self._add_bias:
          # The bias is added via a constant 1 feature.
          self._global_context_dim += 1
      self._overall_context_dim = (
          self._global_context_dim + self._arm_context_dim
      )
      cov_matrix_dim = tf.compat.dimension_value(cov_matrix[0].shape[0])
      if self._overall_context_dim != cov_matrix_dim:
          raise ValueError(
              "The dimension of matrix `cov_matrix` must match "
              "overall context dimension {}. "
              "Got {} for `cov_matrix`.".format(
                  self._overall_context_dim, cov_matrix_dim
              )
          )
  
      data_vector_dim = tf.compat.dimension_value(
          data_vector[0].shape[0]
      )
      if self._overall_context_dim != data_vector_dim:
          raise ValueError(
              "The dimension of vector `data_vector` must match "
              "context  dimension {}. "
              "Got {} for `data_vector`.".format(
                  self._overall_context_dim, data_vector_dim
              )
          )
  
      self._dtype = self._data_vector[0].dtype
      self._emit_policy_info = emit_policy_info
      info_spec = self._populate_policy_info_spec(
          time_step_spec.observation,
          observation_and_action_constraint_splitter,
      )
      if theta is not None:
          if self._dtype != theta.dtype:
              raise ValueError(
                  "The dtype of `theta` is expected to be the same as that of "
                  f"`data_vector`: {self._dtype}, but found to be different: "
                  f"{theta.dtype}"
              )
          self._theta = theta
      else:
          self._theta = None
  
      super(LinearBanditPolicy, self).__init__(
          time_step_spec=time_step_spec,
          action_spec=action_spec,
          info_spec=info_spec,
          emit_log_probability=emit_log_probability,
          observation_and_action_constraint_splitter=(
              observation_and_action_constraint_splitter
          ),
          name=name,
      )
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  import enum
  from typing import Optional, Sequence, Text, Tuple
  import tensorflow
  import tensorflow_probability
  from tf_agents.bandits.policies import constraints
  from tf_agents.bandits.policies import linalg
  from tf_agents.bandits.specs import utils
  from tf_agents.policies import tf_policy
  from tf_agents.policies import utils
  from tf_agents.specs import tensor_spec
  from tf_agents.trajectories import policy_step
  from tf_agents.typing import types
label_desc: Use image node_id 1 to create a new LinearBanditPolicy object from inherited base classes: tf_policy with example: obj = LinearBanditPolicy(action_spec, cov_matrix, data_vector, num_samples, time_step_spec, exploration_strategy, alpha, eig_vals, eig_matrix, tikhonov_weight, add_bias, emit_policy_info, emit_log_probability, accepts_per_arm_features, observation_and_action_constraint_splitter, theta, name)