class_docstr: Generator module in VITS.
  
  This is a module of VITS described in `Conditional Variational Autoencoder
  with Adversarial Learning for End-to-End Text-to-Speech`_.
  
  As text encoder, we use conformer architecture instead of the relative positional
  Transformer, which contains additional convolution layers.
  
  .. _`Conditional Variational Autoencoder with Adversarial Learning for End-to-End
      Text-to-Speech`: https://arxiv.org/abs/2006.04558
class_docstr_tok: ["Generator","module","in","VITS",".","This","is","a","module","of","VITS","described","in","`","Conditional","Variational","Autoencoder","with","Adversarial","Learning","for","End-to-End","Text-to-Speech","`","_",".","As","text","encoder",",","we","use","conformer","architecture","instead","of","the","relative","positional","Transformer",",","which","contains","additional","convolution","layers",".","..","_","`","Conditional","Variational","Autoencoder","with","Adversarial","Learning","for","End-to-End","Text-to-Speech","`",":","https",":","\/\/arxiv.org\/abs\/2006.04558"]
docstr: Initialize VITS generator module.
  
  Args:
      vocabs (int): Input vocabulary size.
      aux_channels (int): Number of acoustic feature channels.
      hidden_channels (int): Number of hidden channels.
      spks (Optional[int]): Number of speakers. If set to > 1, assume that the
          sids will be provided as the input and use sid embedding layer.
      langs (Optional[int]): Number of languages. If set to > 1, assume that the
          lids will be provided as the input and use sid embedding layer.
      spk_embed_dim (Optional[int]): Speaker embedding dimension. If set to > 0,
          assume that spembs will be provided as the input.
      global_channels (int): Number of global conditioning channels.
      segment_size (int): Segment size for decoder.
      text_encoder_attention_heads (int): Number of heads in conformer block
          of text encoder.
      text_encoder_ffn_expand (int): Expansion ratio of FFN in conformer block
          of text encoder.
      text_encoder_blocks (int): Number of conformer blocks in text encoder.
      text_encoder_positionwise_layer_type (str): Position-wise layer type in
          conformer block of text encoder.
      text_encoder_positionwise_conv_kernel_size (int): Position-wise convolution
          kernel size in conformer block of text encoder. Only used when the
          above layer type is conv1d or conv1d-linear.
      text_encoder_positional_encoding_layer_type (str): Positional encoding layer
          type in conformer block of text encoder.
      text_encoder_self_attention_layer_type (str): Self-attention layer type in
          conformer block of text encoder.
      text_encoder_activation_type (str): Activation function type in conformer
          block of text encoder.
      text_encoder_normalize_before (bool): Whether to apply layer norm before
          self-attention in conformer block of text encoder.
      text_encoder_dropout_rate (float): Dropout rate in conformer block of
          text encoder.
      text_encoder_positional_dropout_rate (float): Dropout rate for positional
          encoding in conformer block of text encoder.
      text_encoder_attention_dropout_rate (float): Dropout rate for attention in
          conformer block of text encoder.
      text_encoder_conformer_kernel_size (int): Conformer conv kernel size. It
          will be used when only use_conformer_conv_in_text_encoder = True.
      use_macaron_style_in_text_encoder (bool): Whether to use macaron style FFN
          in conformer block of text encoder.
      use_conformer_conv_in_text_encoder (bool): Whether to use covolution in
          conformer block of text encoder.
      decoder_kernel_size (int): Decoder kernel size.
      decoder_channels (int): Number of decoder initial channels.
      decoder_upsample_scales (List[int]): List of upsampling scales in decoder.
      decoder_upsample_kernel_sizes (List[int]): List of kernel size for
          upsampling layers in decoder.
      decoder_resblock_kernel_sizes (List[int]): List of kernel size for resblocks
          in decoder.
      decoder_resblock_dilations (List[List[int]]): List of list of dilations for
          resblocks in decoder.
      use_weight_norm_in_decoder (bool): Whether to apply weight normalization in
          decoder.
      posterior_encoder_kernel_size (int): Posterior encoder kernel size.
      posterior_encoder_layers (int): Number of layers of posterior encoder.
      posterior_encoder_stacks (int): Number of stacks of posterior encoder.
      posterior_encoder_base_dilation (int): Base dilation of posterior encoder.
      posterior_encoder_dropout_rate (float): Dropout rate for posterior encoder.
      use_weight_norm_in_posterior_encoder (bool): Whether to apply weight
          normalization in posterior encoder.
      flow_flows (int): Number of flows in flow.
      flow_kernel_size (int): Kernel size in flow.
      flow_base_dilation (int): Base dilation in flow.
      flow_layers (int): Number of layers in flow.
      flow_dropout_rate (float): Dropout rate in flow
      use_weight_norm_in_flow (bool): Whether to apply weight normalization in
          flow.
      use_only_mean_in_flow (bool): Whether to use only mean in flow.
      stochastic_duration_predictor_kernel_size (int): Kernel size in stochastic
          duration predictor.
      stochastic_duration_predictor_dropout_rate (float): Dropout rate in
          stochastic duration predictor.
      stochastic_duration_predictor_flows (int): Number of flows in stochastic
          duration predictor.
      stochastic_duration_predictor_dds_conv_layers (int): Number of DDS conv
          layers in stochastic duration predictor.
docstr_tok: ["Initialize","VITS","generator","module",".","Args",":","vocabs","(","int",")",":","Input","vocabulary","size",".","aux_channels","(","int",")",":","Number","of","acoustic","feature","channels",".","hidden_channels","(","int",")",":","Number","of","hidden","channels",".","spks","(","Optional","[","int","]",")",":","Number","of","speakers",".","If","set","to",">","1",",","assume","that","the","sids","will","be","provided","as","the","input","and","use","sid","embedding","layer",".","langs","(","Optional","[","int","]",")",":","Number","of","languages",".","If","set","to",">","1",",","assume","that","the","lids","will","be","provided","as","the","input","and","use","sid","embedding","layer",".","spk_embed_dim","(","Optional","[","int","]",")",":","Speaker","embedding","dimension",".","If","set","to",">","0",",","assume","that","spembs","will","be","provided","as","the","input",".","global_channels","(","int",")",":","Number","of","global","conditioning","channels",".","segment_size","(","int",")",":","Segment","size","for","decoder",".","text_encoder_attention_heads","(","int",")",":","Number","of","heads","in","conformer","block","of","text","encoder",".","text_encoder_ffn_expand","(","int",")",":","Expansion","ratio","of","FFN","in","conformer","block","of","text","encoder",".","text_encoder_blocks","(","int",")",":","Number","of","conformer","blocks","in","text","encoder",".","text_encoder_positionwise_layer_type","(","str",")",":","Position-wise","layer","type","in","conformer","block","of","text","encoder",".","text_encoder_positionwise_conv_kernel_size","(","int",")",":","Position-wise","convolution","kernel","size","in","conformer","block","of","text","encoder",".","Only","used","when","the","above","layer","type","is","conv1d","or","conv1d-linear",".","text_encoder_positional_encoding_layer_type","(","str",")",":","Positional","encoding","layer","type","in","conformer","block","of","text","encoder",".","text_encoder_self_attention_layer_type","(","str",")",":","Self-attention","layer","type","in","conformer","block","of","text","encoder",".","text_encoder_activation_type","(","str",")",":","Activation","function","type","in","conformer","block","of","text","encoder",".","text_encoder_normalize_before","(","bool",")",":","Whether","to","apply","layer","norm","before","self-attention","in","conformer","block","of","text","encoder",".","text_encoder_dropout_rate","(","float",")",":","Dropout","rate","in","conformer","block","of","text","encoder",".","text_encoder_positional_dropout_rate","(","float",")",":","Dropout","rate","for","positional","encoding","in","conformer","block","of","text","encoder",".","text_encoder_attention_dropout_rate","(","float",")",":","Dropout","rate","for","attention","in","conformer","block","of","text","encoder",".","text_encoder_conformer_kernel_size","(","int",")",":","Conformer","conv","kernel","size",".","It","will","be","used","when","only","use_conformer_conv_in_text_encoder","=","True",".","use_macaron_style_in_text_encoder","(","bool",")",":","Whether","to","use","macaron","style","FFN","in","conformer","block","of","text","encoder",".","use_conformer_conv_in_text_encoder","(","bool",")",":","Whether","to","use","covolution","in","conformer","block","of","text","encoder",".","decoder_kernel_size","(","int",")",":","Decoder","kernel","size",".","decoder_channels","(","int",")",":","Number","of","decoder","initial","channels",".","decoder_upsample_scales","(","List","[","int","]",")",":","List","of","upsampling","scales","in","decoder",".","decoder_upsample_kernel_sizes","(","List","[","int","]",")",":","List","of","kernel","size","for","upsampling","layers","in","decoder",".","decoder_resblock_kernel_sizes","(","List","[","int","]",")",":","List","of","kernel","size","for","resblocks","in","decoder",".","decoder_resblock_dilations","(","List","[","List","[","int","]","]",")",":","List","of","list","of","dilations","for","resblocks","in","decoder",".","use_weight_norm_in_decoder","(","bool",")",":","Whether","to","apply","weight","normalization","in","decoder",".","posterior_encoder_kernel_size","(","int",")",":","Posterior","encoder","kernel","size",".","posterior_encoder_layers","(","int",")",":","Number","of","layers","of","posterior","encoder",".","posterior_encoder_stacks","(","int",")",":","Number","of","stacks","of","posterior","encoder",".","posterior_encoder_base_dilation","(","int",")",":","Base","dilation","of","posterior","encoder",".","posterior_encoder_dropout_rate","(","float",")",":","Dropout","rate","for","posterior","encoder",".","use_weight_norm_in_posterior_encoder","(","bool",")",":","Whether","to","apply","weight","normalization","in","posterior","encoder",".","flow_flows","(","int",")",":","Number","of","flows","in","flow",".","flow_kernel_size","(","int",")",":","Kernel","size","in","flow",".","flow_base_dilation","(","int",")",":","Base","dilation","in","flow",".","flow_layers","(","int",")",":","Number","of","layers","in","flow",".","flow_dropout_rate","(","float",")",":","Dropout","rate","in","flow","use_weight_norm_in_flow","(","bool",")",":","Whether","to","apply","weight","normalization","in","flow",".","use_only_mean_in_flow","(","bool",")",":","Whether","to","use","only","mean","in","flow",".","stochastic_duration_predictor_kernel_size","(","int",")",":","Kernel","size","in","stochastic","duration","predictor",".","stochastic_duration_predictor_dropout_rate","(","float",")",":","Dropout","rate","in","stochastic","duration","predictor",".","stochastic_duration_predictor_flows","(","int",")",":","Number","of","flows","in","stochastic","duration","predictor",".","stochastic_duration_predictor_dds_conv_layers","(","int",")",":","Number","of","DDS","conv","layers","in","stochastic","duration","predictor","."]
code: def __init__(
      self,
      vocabs: int,
      aux_channels: int = 513,
      hidden_channels: int = 192,
      spks: Optional[int] = None,
      langs: Optional[int] = None,
      spk_embed_dim: Optional[int] = None,
      global_channels: int = -1,
      segment_size: int = 32,
      text_encoder_attention_heads: int = 2,
      text_encoder_ffn_expand: int = 4,
      text_encoder_blocks: int = 6,
      text_encoder_positionwise_layer_type: str = "conv1d",
      text_encoder_positionwise_conv_kernel_size: int = 1,
      text_encoder_positional_encoding_layer_type: str = "rel_pos",
      text_encoder_self_attention_layer_type: str = "rel_selfattn",
      text_encoder_activation_type: str = "swish",
      text_encoder_normalize_before: bool = True,
      text_encoder_dropout_rate: float = 0.1,
      text_encoder_positional_dropout_rate: float = 0.0,
      text_encoder_attention_dropout_rate: float = 0.0,
      text_encoder_conformer_kernel_size: int = 7,
      use_macaron_style_in_text_encoder: bool = True,
      use_conformer_conv_in_text_encoder: bool = True,
      decoder_kernel_size: int = 7,
      decoder_channels: int = 512,
      decoder_upsample_scales: List[int] = [8, 8, 2, 2],
      decoder_upsample_kernel_sizes: List[int] = [16, 16, 4, 4],
      decoder_resblock_kernel_sizes: List[int] = [3, 7, 11],
      decoder_resblock_dilations: List[List[int]] = [
          [1, 3, 5],
          [1, 3, 5],
          [1, 3, 5],
      ],
      use_weight_norm_in_decoder: bool = True,
      posterior_encoder_kernel_size: int = 5,
      posterior_encoder_layers: int = 16,
      posterior_encoder_stacks: int = 1,
      posterior_encoder_base_dilation: int = 1,
      posterior_encoder_dropout_rate: float = 0.0,
      use_weight_norm_in_posterior_encoder: bool = True,
      flow_flows: int = 4,
      flow_kernel_size: int = 5,
      flow_base_dilation: int = 1,
      flow_layers: int = 4,
      flow_dropout_rate: float = 0.0,
      use_weight_norm_in_flow: bool = True,
      use_only_mean_in_flow: bool = True,
      stochastic_duration_predictor_kernel_size: int = 3,
      stochastic_duration_predictor_dropout_rate: float = 0.5,
      stochastic_duration_predictor_flows: int = 4,
      stochastic_duration_predictor_dds_conv_layers: int = 3,
  ):
      """Initialize VITS generator module.
  
      Args:
          vocabs (int): Input vocabulary size.
          aux_channels (int): Number of acoustic feature channels.
          hidden_channels (int): Number of hidden channels.
          spks (Optional[int]): Number of speakers. If set to > 1, assume that the
              sids will be provided as the input and use sid embedding layer.
          langs (Optional[int]): Number of languages. If set to > 1, assume that the
              lids will be provided as the input and use sid embedding layer.
          spk_embed_dim (Optional[int]): Speaker embedding dimension. If set to > 0,
              assume that spembs will be provided as the input.
          global_channels (int): Number of global conditioning channels.
          segment_size (int): Segment size for decoder.
          text_encoder_attention_heads (int): Number of heads in conformer block
              of text encoder.
          text_encoder_ffn_expand (int): Expansion ratio of FFN in conformer block
              of text encoder.
          text_encoder_blocks (int): Number of conformer blocks in text encoder.
          text_encoder_positionwise_layer_type (str): Position-wise layer type in
              conformer block of text encoder.
          text_encoder_positionwise_conv_kernel_size (int): Position-wise convolution
              kernel size in conformer block of text encoder. Only used when the
              above layer type is conv1d or conv1d-linear.
          text_encoder_positional_encoding_layer_type (str): Positional encoding layer
              type in conformer block of text encoder.
          text_encoder_self_attention_layer_type (str): Self-attention layer type in
              conformer block of text encoder.
          text_encoder_activation_type (str): Activation function type in conformer
              block of text encoder.
          text_encoder_normalize_before (bool): Whether to apply layer norm before
              self-attention in conformer block of text encoder.
          text_encoder_dropout_rate (float): Dropout rate in conformer block of
              text encoder.
          text_encoder_positional_dropout_rate (float): Dropout rate for positional
              encoding in conformer block of text encoder.
          text_encoder_attention_dropout_rate (float): Dropout rate for attention in
              conformer block of text encoder.
          text_encoder_conformer_kernel_size (int): Conformer conv kernel size. It
              will be used when only use_conformer_conv_in_text_encoder = True.
          use_macaron_style_in_text_encoder (bool): Whether to use macaron style FFN
              in conformer block of text encoder.
          use_conformer_conv_in_text_encoder (bool): Whether to use covolution in
              conformer block of text encoder.
          decoder_kernel_size (int): Decoder kernel size.
          decoder_channels (int): Number of decoder initial channels.
          decoder_upsample_scales (List[int]): List of upsampling scales in decoder.
          decoder_upsample_kernel_sizes (List[int]): List of kernel size for
              upsampling layers in decoder.
          decoder_resblock_kernel_sizes (List[int]): List of kernel size for resblocks
              in decoder.
          decoder_resblock_dilations (List[List[int]]): List of list of dilations for
              resblocks in decoder.
          use_weight_norm_in_decoder (bool): Whether to apply weight normalization in
              decoder.
          posterior_encoder_kernel_size (int): Posterior encoder kernel size.
          posterior_encoder_layers (int): Number of layers of posterior encoder.
          posterior_encoder_stacks (int): Number of stacks of posterior encoder.
          posterior_encoder_base_dilation (int): Base dilation of posterior encoder.
          posterior_encoder_dropout_rate (float): Dropout rate for posterior encoder.
          use_weight_norm_in_posterior_encoder (bool): Whether to apply weight
              normalization in posterior encoder.
          flow_flows (int): Number of flows in flow.
          flow_kernel_size (int): Kernel size in flow.
          flow_base_dilation (int): Base dilation in flow.
          flow_layers (int): Number of layers in flow.
          flow_dropout_rate (float): Dropout rate in flow
          use_weight_norm_in_flow (bool): Whether to apply weight normalization in
              flow.
          use_only_mean_in_flow (bool): Whether to use only mean in flow.
          stochastic_duration_predictor_kernel_size (int): Kernel size in stochastic
              duration predictor.
          stochastic_duration_predictor_dropout_rate (float): Dropout rate in
              stochastic duration predictor.
          stochastic_duration_predictor_flows (int): Number of flows in stochastic
              duration predictor.
          stochastic_duration_predictor_dds_conv_layers (int): Number of DDS conv
              layers in stochastic duration predictor.
  
      """
      super().__init__()
      self.segment_size = segment_size
      self.text_encoder = TextEncoder(
          vocabs=vocabs,
          attention_dim=hidden_channels,
          attention_heads=text_encoder_attention_heads,
          linear_units=hidden_channels * text_encoder_ffn_expand,
          blocks=text_encoder_blocks,
          positionwise_layer_type=text_encoder_positionwise_layer_type,
          positionwise_conv_kernel_size=text_encoder_positionwise_conv_kernel_size,
          positional_encoding_layer_type=text_encoder_positional_encoding_layer_type,
          self_attention_layer_type=text_encoder_self_attention_layer_type,
          activation_type=text_encoder_activation_type,
          normalize_before=text_encoder_normalize_before,
          dropout_rate=text_encoder_dropout_rate,
          positional_dropout_rate=text_encoder_positional_dropout_rate,
          attention_dropout_rate=text_encoder_attention_dropout_rate,
          conformer_kernel_size=text_encoder_conformer_kernel_size,
          use_macaron_style=use_macaron_style_in_text_encoder,
          use_conformer_conv=use_conformer_conv_in_text_encoder,
      )
      self.decoder = HiFiGANGenerator(
          in_channels=hidden_channels,
          out_channels=1,
          channels=decoder_channels,
          global_channels=global_channels,
          kernel_size=decoder_kernel_size,
          upsample_scales=decoder_upsample_scales,
          upsample_kernel_sizes=decoder_upsample_kernel_sizes,
          resblock_kernel_sizes=decoder_resblock_kernel_sizes,
          resblock_dilations=decoder_resblock_dilations,
          use_weight_norm=use_weight_norm_in_decoder,
      )
      self.posterior_encoder = PosteriorEncoder(
          in_channels=aux_channels,
          out_channels=hidden_channels,
          hidden_channels=hidden_channels,
          kernel_size=posterior_encoder_kernel_size,
          layers=posterior_encoder_layers,
          stacks=posterior_encoder_stacks,
          base_dilation=posterior_encoder_base_dilation,
          global_channels=global_channels,
          dropout_rate=posterior_encoder_dropout_rate,
          use_weight_norm=use_weight_norm_in_posterior_encoder,
      )
      self.flow = ResidualAffineCouplingBlock(
          in_channels=hidden_channels,
          hidden_channels=hidden_channels,
          flows=flow_flows,
          kernel_size=flow_kernel_size,
          base_dilation=flow_base_dilation,
          layers=flow_layers,
          global_channels=global_channels,
          dropout_rate=flow_dropout_rate,
          use_weight_norm=use_weight_norm_in_flow,
          use_only_mean=use_only_mean_in_flow,
      )
      # TODO(kan-bayashi): Add deterministic version as an option
      self.duration_predictor = StochasticDurationPredictor(
          channels=hidden_channels,
          kernel_size=stochastic_duration_predictor_kernel_size,
          dropout_rate=stochastic_duration_predictor_dropout_rate,
          flows=stochastic_duration_predictor_flows,
          dds_conv_layers=stochastic_duration_predictor_dds_conv_layers,
          global_channels=global_channels,
      )
  
      self.upsample_factor = int(np.prod(decoder_upsample_scales))
      self.spks = None
      if spks is not None and spks > 1:
          assert global_channels > 0
          self.spks = spks
          self.global_emb = torch.nn.Embedding(spks, global_channels)
      self.spk_embed_dim = None
      if spk_embed_dim is not None and spk_embed_dim > 0:
          assert global_channels > 0
          self.spk_embed_dim = spk_embed_dim
          self.spemb_proj = torch.nn.Linear(
              spk_embed_dim, global_channels
          )
      self.langs = None
      if langs is not None and langs > 1:
          assert global_channels > 0
          self.langs = langs
          self.lang_emb = torch.nn.Embedding(langs, global_channels)
  
      # delayed import
      from espnet2.gan_tts.vits.monotonic_align import maximum_path
  
      self.maximum_path = maximum_path
  
file_path: espnet/espnet2/gan_tts/vits/generator.py
imports: import math
  from typing import List, Optional, Tuple
  import numpy
  import torch
  import torch.nn.functional
  from espnet2.gan_tts.hifigan import HiFiGANGenerator
  from espnet2.gan_tts.utils import get_random_segments
  from espnet2.gan_tts.vits.duration_predictor import StochasticDurationPredictor
  from espnet2.gan_tts.vits.posterior_encoder import PosteriorEncoder
  from espnet2.gan_tts.vits.residual_coupling import ResidualAffineCouplingBlock
  from espnet2.gan_tts.vits.text_encoder import TextEncoder
  from espnet.nets.pytorch_backend.nets_utils import make_non_pad_mask
label_desc: Use image node_id 1 to create a new VITSGenerator object from inherited base classes: torch.nn with example: obj = VITSGenerator(vocabs, aux_channels, hidden_channels, spks, langs, spk_embed_dim, global_channels, segment_size, text_encoder_attention_heads, text_encoder_ffn_expand, text_encoder_blocks, text_encoder_positionwise_layer_type, text_encoder_positionwise_conv_kernel_size, text_encoder_positional_encoding_layer_type, text_encoder__attention_layer_type, text_encoder_activation_type, text_encoder_normalize_before, text_encoder_dropout_rate, text_encoder_positional_dropout_rate, text_encoder_attention_dropout_rate, text_encoder_conformer_kernel_size, use_macaron_style_in_text_encoder, use_conformer_conv_in_text_encoder, decoder_kernel_size, decoder_channels, decoder_upsample_scales, decoder_upsample_kernel_sizes, decoder_resblock_kernel_sizes, decoder_resblock_dilations, use_weight_norm_in_decoder, posterior_encoder_kernel_size, posterior_encoder_layers, posterior_encoder_stacks, posterior_encoder_base_dilation, posterior_encoder_dropout_rate, use_weight_norm_in_posterior_encoder, flow_flows, flow_kernel_size, flow_base_dilation, flow_layers, flow_dropout_rate, use_weight_norm_in_flow, use_only_mean_in_flow, stochastic_duration_predictor_kernel_size, stochastic_duration_predictor_dropout_rate, stochastic_duration_predictor_flows, stochastic_duration_predictor_dds_conv_layers)