class_docstr: Constructes a diagonal Gaussian distribution from the hidden states and samples from it using reparametrization.
class_docstr_tok: ["Constructes","a","diagonal","Gaussian","distribution","from","the","hidden","states","and","samples","from","it","using","reparametrization","."]
code: def __init__(
      self,
      hidden_size,
      ffn_hidden_size=None,
      min_logvar=-6,
      init_method_std=0.02,
      name="cond_gaussian",
      model_parallel_cfg: ModelParallelConfig = None,
  ):
      super().__init__(name=name, model_parallel_cfg=model_parallel_cfg)
      # limit smaller allowed variance (for numerical stability)
      self.min_logvar = min_logvar
      self.hidden_size = hidden_size
      if ffn_hidden_size is None:
          ffn_hidden_size = hidden_size * 2
      self.ffn_hidden_size = ffn_hidden_size
  
      # project hiddens to mean and log variance (support tensor parallelism)
      self.hiddens_to_mean_logvar = tensor_parallel.ColumnParallelLinear(
          hidden_size,
          ffn_hidden_size,  # NOTE: When using *glu, divide ffn dim by 2/3 to keep overall params the same.
          gather_output=True,
          init_method=init_method_normal(init_method_std),
          skip_bias_add=False,
          bias=True,
          config=self.model_parallel_cfg,
      )
  
file_path: NeMo/nemo/collections/nlp/modules/common/megatron/hiddens/megatron_hidden_transform.py
imports: import math
  import torch
  from nemo.collections.nlp.modules.common.megatron.utils import ApexGuardDefaults, init_method_normal
label_desc: Use image node_id 1 to create a new MegatronGaussianHiddenTransform object from inherited base classes: MegatronBaseHiddenTransform with example: obj = MegatronGaussianHiddenTransform(hidden_size, ffn_hidden_size, min_logvar, init_method_std, name, model_parallel_cfg)