class_docstr: Densely-connected layer class with reparameterization estimator.
  
  This layer implements the Bayesian variational inference analogue to
  a dense layer by assuming the `kernel` and/or the `bias` are drawn
  from distributions. By default, the layer implements a stochastic
  forward pass via sampling from the kernel and bias posteriors,
  
  ```none
  kernel, bias ~ posterior
  outputs = matmul(inputs, kernel) + bias
  ```
  
  It uses the reparameterization estimator [(Kingma and Welling, 2014)][1],
  which performs a Monte Carlo approximation of the distribution integrating
  over the `kernel` and `bias`.
  
  The arguments permit separate specification of the surrogate posterior
  (`q(W|x)`), prior (`p(W)`), and divergence for both the `kernel` and `bias`
  distributions.
  
  Upon being built, this layer adds losses (accessible via the `losses`
  property) representing the divergences of `kernel` and/or `bias` surrogate
  posteriors and their respective priors. When doing minibatch stochastic
  optimization, make sure to scale this loss such that it is applied just once
  per epoch (e.g. if `kl` is the sum of `losses` for each element of the batch,
  you should pass `kl / num_examples_per_epoch` to your optimizer).
  
  You can access the `kernel` and/or `bias` posterior and prior distributions
  after the layer is built via the `kernel_posterior`, `kernel_prior`,
  `bias_posterior` and `bias_prior` properties.
  
  #### Examples
  
  We illustrate a Bayesian neural network with [variational inference](
  https://en.wikipedia.org/wiki/Variational_Bayesian_methods),
  assuming a dataset of images and length-10 one-hot `targets`.
  
  ```python
  import functools
  import tensorflow.compat.v2 as tf
  import tensorflow_probability as tfp
  import tensorflow_datasets as tfds
  tfb = tfp.bijectors
  tfd = tfp.distributions
  tfn = tfp.experimental.nn
  
  # 1  Prepare Dataset
  
  [train_dataset, eval_dataset], datasets_info = tfds.load(
      name='mnist',
      split=['train', 'test'],
      with_info=True,
      as_supervised=True,
      shuffle_files=True)
  def _preprocess(image, label):
    # image = image < tf.random.uniform(tf.shape(image))   # Randomly binarize.
    image = tf.cast(image, tf.float32) / 255.  # Scale to unit interval.
    lo = 0.001
    image = (1. - 2. * lo) * image + lo  # Rescale to *open* unit interval.
    return image, label
  batch_size = 32
  train_size = datasets_info.splits['train'].num_examples
  train_dataset = tfn.util.tune_dataset(
      train_dataset,
      batch_shape=(batch_size,),
      shuffle_size=int(train_size / 7),
      preprocess_fn=_preprocess)
  train_iter = iter(train_dataset)
  eval_iter = iter(eval_dataset)
  x, y = next(train_iter)
  evidence_shape = x.shape[1:]
  targets_shape = y.shape[1:]
  
  # 2  Specify Model
  
  BayesConv2D = functools.partial(
      tfn.ConvolutionVariationalReparameterization,
      rank=2,
      padding='same',
      filter_shape=5,
      # Use `he_uniform` because we'll use the `relu` family.
      kernel_initializer=tf_keras.initializers.he_uniform())
  
  BayesAffine = functools.partial(
      tfn.AffineVariationalReparameterization,
      kernel_initializer=tf_keras.initializers.he_normal())
  
  scale = tfp.util.TransformedVariable(1., tfb.Softplus())
  bnn = tfn.Sequential([
      BayesConv2D(evidence_shape[-1], 32, filter_shape=7, strides=2,
                  activation_fn=tf.nn.leaky_relu),           # [b, 14, 14, 32]
      tfn.util.flatten_rightmost(ndims=3),                   # [b, 14 * 14 * 32]
      BayesAffine(14 * 14 * 32, np.prod(target_shape) - 1),  # [b, 9]
      lambda loc: tfb.SoftmaxCentered()(
          tfd.Independent(tfd.Normal(loc, scale),
                          reinterpreted_batch_ndims=1))  # [b, 10]
  ], name='bayesian_neural_network')
  
  print(bnn.summary())
  
  # 3  Train.
  
  def loss_fn():
    x, y = next(train_iter)
    nll = -tf.reduce_mean(bnn(x).log_prob(y), axis=-1)
    kl = bnn.extra_loss / tf.cast(train_size, tf.float32)
    loss = nll + kl
    return loss, (nll, kl)
  opt = tf_keras.optimizers.Adam()
  fit_op = tfn.util.make_fit_op(loss_fn, opt, bnn.trainable_variables)
  for _ in range(200):
    loss, (nll, kl), g = fit_op()
  ```
  
  This example uses reparameterization gradients to minimize the
  Kullback-Leibler divergence up to a constant, also known as the negative
  Evidence Lower Bound. It consists of the sum of two terms: the expected
  negative log-likelihood, which we approximate via Monte Carlo; and the KL
  divergence, which is added via regularizer terms which are arguments to the
  layer.
  
  #### References
  
  [1]: Diederik Kingma and Max Welling. Auto-Encoding Variational Bayes. In
       _International Conference on Learning Representations_, 2014.
       https://arxiv.org/abs/1312.6114
class_docstr_tok: ["Densely-connected","layer","class","with","reparameterization","estimator",".","This","layer","implements","the","Bayesian","variational","inference","analogue","to","a","dense","layer","by","assuming","the","`","kernel","`","and\/or","the","`","bias","`","are","drawn","from","distributions",".","By","default",",","the","layer","implements","a","stochastic","forward","pass","via","sampling","from","the","kernel","and","bias","posteriors",",","``","`","none","kernel",",","bias","~","posterior","outputs","=","matmul","(","inputs",",","kernel",")","+","bias","``","`","It","uses","the","reparameterization","estimator","[","(","Kingma","and","Welling",",","2014",")","]","[","1","]",",","which","performs","a","Monte","Carlo","approximation","of","the","distribution","integrating","over","the","`","kernel","`","and","`","bias","`",".","The","arguments","permit","separate","specification","of","the","surrogate","posterior","(","`","q","(","W|x",")","`",")",",","prior","(","`","p","(","W",")","`",")",",","and","divergence","for","both","the","`","kernel","`","and","`","bias","`","distributions",".","Upon","being","built",",","this","layer","adds","losses","(","accessible","via","the","`","losses","`","property",")","representing","the","divergences","of","`","kernel","`","and\/or","`","bias","`","surrogate","posteriors","and","their","respective","priors",".","When","doing","minibatch","stochastic","optimization",",","make","sure","to","scale","this","loss","such","that","it","is","applied","just","once","per","epoch","(","e.g",".","if","`","kl","`","is","the","sum","of","`","losses","`","for","each","element","of","the","batch",",","you","should","pass","`","kl","\/","num_examples_per_epoch","`","to","your","optimizer",")",".","You","can","access","the","`","kernel","`","and\/or","`","bias","`","posterior","and","prior","distributions","after","the","layer","is","built","via","the","`","kernel_posterior","`",",","`","kernel_prior","`",",","`","bias_posterior","`","and","`","bias_prior","`","properties",".","#","#","#","#","Examples","We","illustrate","a","Bayesian","neural","network","with","[","variational","inference","]","(","https",":","\/\/en.wikipedia.org\/wiki\/Variational_Bayesian_methods",")",",","assuming","a","dataset","of","images","and","length-10","one-hot","`","targets","`",".","``","`","python","import","functools","import","tensorflow.compat.v2","as","tf","import","tensorflow_probability","as","tfp","import","tensorflow_datasets","as","tfds","tfb","=","tfp.bijectors","tfd","=","tfp.distributions","tfn","=","tfp.experimental.nn","#","1","Prepare","Dataset","[","train_dataset",",","eval_dataset","]",",","datasets_info","=","tfds.load","(","name='mnist","'",",","split=","[","'train","'",",","'test","'","]",",","with_info=True",",","as_supervised=True",",","shuffle_files=True",")","def","_preprocess","(","image",",","label",")",":","#","image","=","image","<","tf.random.uniform","(","tf.shape","(","image",")",")","#","Randomly","binarize",".","image","=","tf.cast","(","image",",","tf.float32",")","\/","255",".","#","Scale","to","unit","interval",".","lo","=","0.001","image","=","(","1",".","-","2",".","*","lo",")","*","image","+","lo","#","Rescale","to","*","open","*","unit","interval",".","return","image",",","label","batch_size","=","32","train_size","=","datasets_info.splits","[","'train","'","]",".num_examples","train_dataset","=","tfn.util.tune_dataset","(","train_dataset",",","batch_shape=","(","batch_size",",",")",",","shuffle_size=int","(","train_size","\/","7",")",",","preprocess_fn=_preprocess",")","train_iter","=","iter","(","train_dataset",")","eval_iter","=","iter","(","eval_dataset",")","x",",","y","=","next","(","train_iter",")","evidence_shape","=","x.shape","[","1",":","]","targets_shape","=","y.shape","[","1",":","]","#","2","Specify","Model","BayesConv2D","=","functools.partial","(","tfn.ConvolutionVariationalReparameterization",",","rank=2",",","padding='same","'",",","filter_shape=5",",","#","Use","`","he_uniform","`","because","we","'ll","use","the","`","relu","`","family",".","kernel_initializer=tf_keras.initializers.he_uniform","(",")",")","BayesAffine","=","functools.partial","(","tfn.AffineVariationalReparameterization",",","kernel_initializer=tf_keras.initializers.he_normal","(",")",")","scale","=","tfp.util.TransformedVariable","(","1.",",","tfb.Softplus","(",")",")","bnn","=","tfn.Sequential","(","[","BayesConv2D","(","evidence_shape","[","-1","]",",","32",",","filter_shape=7",",","strides=2",",","activation_fn=tf.nn.leaky_relu",")",",","#","[","b",",","14",",","14",",","32","]","tfn.util.flatten_rightmost","(","ndims=3",")",",","#","[","b",",","14","*","14","*","32","]","BayesAffine","(","14","*","14","*","32",",","np.prod","(","target_shape",")","-","1",")",",","#","[","b",",","9","]","lambda","loc",":","tfb.SoftmaxCentered","(",")","(","tfd.Independent","(","tfd.Normal","(","loc",",","scale",")",",","reinterpreted_batch_ndims=1",")",")","#","[","b",",","10","]","]",",","name='bayesian_neural_network","'",")","print","(","bnn.summary","(",")",")","#","3","Train",".","def","loss_fn","(",")",":","x",",","y","=","next","(","train_iter",")","nll","=","-tf.reduce_mean","(","bnn","(","x",")",".log_prob","(","y",")",",","axis=-1",")","kl","=","bnn.extra_loss","\/","tf.cast","(","train_size",",","tf.float32",")","loss","=","nll","+","kl","return","loss",",","(","nll",",","kl",")","opt","=","tf_keras.optimizers.Adam","(",")","fit_op","=","tfn.util.make_fit_op","(","loss_fn",",","opt",",","bnn.trainable_variables",")","for","_","in","range","(","200",")",":","loss",",","(","nll",",","kl",")",",","g","=","fit_op","(",")","``","`","This","example","uses","reparameterization","gradients","to","minimize","the","Kullback-Leibler","divergence","up","to","a","constant",",","also","known","as","the","negative","Evidence","Lower","Bound",".","It","consists","of","the","sum","of","two","terms",":","the","expected","negative","log-likelihood",",","which","we","approximate","via","Monte","Carlo",";","and","the","KL","divergence",",","which","is","added","via","regularizer","terms","which","are","arguments","to","the","layer",".","#","#","#","#","References","[","1","]",":","Diederik","Kingma","and","Max","Welling",".","Auto-Encoding","Variational","Bayes",".","In","_International","Conference","on","Learning","Representations_",",","2014.","https",":","\/\/arxiv.org\/abs\/1312.6114"]
docstr: Constructs layer.
  
  Args:
    input_size: ...
    output_size: ...
    kernel_initializer: ...
      Default value: `None` (i.e.,
      `tfp.experimental.nn.initializers.glorot_uniform()`).
    bias_initializer: ...
      Default value: `None` (i.e., `tf_keras.initializers.zeros()`).
    make_posterior_fn: ...
      Default value:
        `tfp.experimental.nn.util.make_kernel_bias_posterior_mvn_diag`.
    make_prior_fn: ...
      Default value:
        `tfp.experimental.nn.util.make_kernel_bias_prior_spike_and_slab`.
    posterior_value_fn: ...
      Default valye: `tfd.Distribution.sample`
    unpack_weights_fn:
      Default value: `unpack_kernel_and_bias`
    dtype: ...
      Default value: `tf.float32`.
    activation_fn: ...
      Default value: `None`.
    seed: ...
      Default value: `None` (i.e., no seed).
    validate_args: ...
      Default value: `False`.
    name: ...
      Default value: `None` (i.e.,
      `'AffineVariationalReparameterization'`).
docstr_tok: ["Constructs","layer",".","Args",":","input_size",":","...","output_size",":","...","kernel_initializer",":","...","Default","value",":","`","None","`","(","i.e.",",","`","tfp.experimental.nn.initializers.glorot_uniform","(",")","`",")",".","bias_initializer",":","...","Default","value",":","`","None","`","(","i.e.",",","`","tf_keras.initializers.zeros","(",")","`",")",".","make_posterior_fn",":","...","Default","value",":","`","tfp.experimental.nn.util.make_kernel_bias_posterior_mvn_diag","`",".","make_prior_fn",":","...","Default","value",":","`","tfp.experimental.nn.util.make_kernel_bias_prior_spike_and_slab","`",".","posterior_value_fn",":","...","Default","valye",":","`","tfd.Distribution.sample","`","unpack_weights_fn",":","Default","value",":","`","unpack_kernel_and_bias","`","dtype",":","...","Default","value",":","`","tf.float32","`",".","activation_fn",":","...","Default","value",":","`","None","`",".","seed",":","...","Default","value",":","`","None","`","(","i.e.",",","no","seed",")",".","validate_args",":","...","Default","value",":","`","False","`",".","name",":","...","Default","value",":","`","None","`","(","i.e.",",","`","'AffineVariationalReparameterization","'","`",")","."]
code: def __init__(
      self,
      input_size,
      output_size,
      # Weights
      kernel_initializer=None,  # tfp.nn.initializers.glorot_uniform()
      bias_initializer=None,  # tf_keras.initializers.zeros()
      make_posterior_fn=kernel_bias_lib.make_kernel_bias_posterior_mvn_diag,
      make_prior_fn=kernel_bias_lib.make_kernel_bias_prior_spike_and_slab,
      posterior_value_fn=tfd.Distribution.sample,
      unpack_weights_fn=unpack_kernel_and_bias,
      dtype=tf.float32,
      # Misc
      activation_fn=None,
      seed=None,
      validate_args=False,
      name=None,
  ):
      """Constructs layer.
  
      Args:
        input_size: ...
        output_size: ...
        kernel_initializer: ...
          Default value: `None` (i.e.,
          `tfp.experimental.nn.initializers.glorot_uniform()`).
        bias_initializer: ...
          Default value: `None` (i.e., `tf_keras.initializers.zeros()`).
        make_posterior_fn: ...
          Default value:
            `tfp.experimental.nn.util.make_kernel_bias_posterior_mvn_diag`.
        make_prior_fn: ...
          Default value:
            `tfp.experimental.nn.util.make_kernel_bias_prior_spike_and_slab`.
        posterior_value_fn: ...
          Default valye: `tfd.Distribution.sample`
        unpack_weights_fn:
          Default value: `unpack_kernel_and_bias`
        dtype: ...
          Default value: `tf.float32`.
        activation_fn: ...
          Default value: `None`.
        seed: ...
          Default value: `None` (i.e., no seed).
        validate_args: ...
          Default value: `False`.
        name: ...
          Default value: `None` (i.e.,
          `'AffineVariationalReparameterization'`).
      """
      self._make_posterior_fn = (
          make_posterior_fn  # For variable tracking.
      )
      self._make_prior_fn = make_prior_fn  # For variable tracking.
      batch_ndims = 0
      super(AffineVariationalReparameterization, self).__init__(
          posterior=make_posterior_fn(
              [input_size, output_size],
              [output_size],
              kernel_initializer,
              bias_initializer,
              batch_ndims,
              batch_ndims,
              dtype,
          ),
          prior=make_prior_fn(
              [input_size, output_size],
              [output_size],
              kernel_initializer,
              bias_initializer,
              batch_ndims,
              batch_ndims,
              dtype,
          ),
          apply_kernel_fn=tf.matmul,
          posterior_value_fn=posterior_value_fn,
          unpack_weights_fn=unpack_weights_fn,
          dtype=dtype,
          activation_fn=activation_fn,
          validate_args=validate_args,
          seed=seed,
          name=name,
      )
  
imports: import tensorflow.compat.v2
  from tensorflow_probability.python.distributions import distribution
  from tensorflow_probability.python.distributions import normal
  from tensorflow_probability.python.experimental.nn import layers
  from tensorflow_probability.python.experimental.nn import variational_base
  from tensorflow_probability.python.experimental.nn.util import kernel_bias
  from tensorflow_probability.python.internal import prefer_static
label_desc: Use image node_id 1 to create a new AffineVariationalReparameterization object from inherited base classes: vi_lib with example: obj = AffineVariationalReparameterization(input_size, output_size, kernel_initializer, bias_initializer, make_posterior_fn, make_prior_fn, posterior_value_fn, unpack_weights_fn, dtype, activation_fn, seed, validate_args, name)