class_docstr: ProDiff module.
  
  This is a module of ProDiff described in `ProDiff: Progressive Fast Diffusion Model
  for High-Quality Text-to-Speech`_.
  
  .. _`ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech`:
      https://arxiv.org/abs/2207.06389
class_docstr_tok: ["ProDiff","module",".","This","is","a","module","of","ProDiff","described","in","`","ProDiff",":","Progressive","Fast","Diffusion","Model","for","High-Quality","Text-to-Speech","`","_",".","..","_","`","ProDiff",":","Progressive","Fast","Diffusion","Model","for","High-Quality","Text-to-Speech","`",":","https",":","\/\/arxiv.org\/abs\/2207.06389"]
docstr: Initialize ProDiff module.
  
  Args:
      idim (int): Dimension of the inputs.
      odim (int): Dimension of the outputs.
      elayers (int): Number of encoder layers.
      eunits (int): Number of encoder hidden units.
      dlayers (int): Number of decoder layers.
      dunits (int): Number of decoder hidden units.
      postnet_layers (int): Number of postnet layers.
      postnet_chans (int): Number of postnet channels.
      postnet_filts (int): Kernel size of postnet.
      postnet_dropout_rate (float): Dropout rate in postnet.
      use_scaled_pos_enc (bool): Whether to use trainable scaled pos encoding.
      use_batch_norm (bool): Whether to use batch normalization in encoder prenet.
      encoder_normalize_before (bool): Whether to apply layernorm layer before
          encoder block.
      decoder_normalize_before (bool): Whether to apply layernorm layer before
          decoder block.
      encoder_concat_after (bool): Whether to concatenate attention layer's input
          and output in encoder.
      decoder_concat_after (bool): Whether to concatenate attention layer's input
          and output in decoder.
      reduction_factor (int): Reduction factor.
      encoder_type (str): Encoder type ("transformer" or "conformer").
      decoder_type (str): Decoder type ("transformer" or "conformer").
      transformer_enc_dropout_rate (float): Dropout rate in encoder except
          attention and positional encoding.
      transformer_enc_positional_dropout_rate (float): Dropout rate after encoder
          positional encoding.
      transformer_enc_attn_dropout_rate (float): Dropout rate in encoder
          self-attention module.
      transformer_dec_dropout_rate (float): Dropout rate in decoder except
          attention & positional encoding.
      transformer_dec_positional_dropout_rate (float): Dropout rate after decoder
          positional encoding.
      transformer_dec_attn_dropout_rate (float): Dropout rate in decoder
          self-attention module.
      conformer_rel_pos_type (str): Relative pos encoding type in conformer.
      conformer_pos_enc_layer_type (str): Pos encoding layer type in conformer.
      conformer_self_attn_layer_type (str): Self-attention layer type in conformer
      conformer_activation_type (str): Activation function type in conformer.
      use_macaron_style_in_conformer: Whether to use macaron style FFN.
      use_cnn_in_conformer: Whether to use CNN in conformer.
      zero_triu: Whether to use zero triu in relative self-attention module.
      conformer_enc_kernel_size: Kernel size of encoder conformer.
      conformer_dec_kernel_size: Kernel size of decoder conformer.
      duration_predictor_layers (int): Number of duration predictor layers.
      duration_predictor_chans (int): Number of duration predictor channels.
      duration_predictor_kernel_size (int): Kernel size of duration predictor.
      duration_predictor_dropout_rate (float): Dropout rate in duration predictor.
      pitch_predictor_layers (int): Number of pitch predictor layers.
      pitch_predictor_chans (int): Number of pitch predictor channels.
      pitch_predictor_kernel_size (int): Kernel size of pitch predictor.
      pitch_predictor_dropout_rate (float): Dropout rate in pitch predictor.
      pitch_embed_kernel_size (float): Kernel size of pitch embedding.
      pitch_embed_dropout_rate (float): Dropout rate for pitch embedding.
      stop_gradient_from_pitch_predictor: Whether to stop gradient from pitch
          predictor to encoder.
      energy_predictor_layers (int): Number of energy predictor layers.
      energy_predictor_chans (int): Number of energy predictor channels.
      energy_predictor_kernel_size (int): Kernel size of energy predictor.
      energy_predictor_dropout_rate (float): Dropout rate in energy predictor.
      energy_embed_kernel_size (float): Kernel size of energy embedding.
      energy_embed_dropout_rate (float): Dropout rate for energy embedding.
      stop_gradient_from_energy_predictor: Whether to stop gradient from energy
          predictor to encoder.
      spks (Optional[int]): Number of speakers. If set to > 1, assume that the
          sids will be provided as the input and use sid embedding layer.
      langs (Optional[int]): Number of languages. If set to > 1, assume that the
          lids will be provided as the input and use sid embedding layer.
      spk_embed_dim (Optional[int]): Speaker embedding dimension. If set to > 0,
          assume that spembs will be provided as the input.
      spk_embed_integration_type: How to integrate speaker embedding.
      use_gst (str): Whether to use global style token.
      gst_tokens (int): The number of GST embeddings.
      gst_heads (int): The number of heads in GST multihead attention.
      gst_conv_layers (int): The number of conv layers in GST.
      gst_conv_chans_list: (Sequence[int]):
          List of the number of channels of conv layers in GST.
      gst_conv_kernel_size (int): Kernel size of conv layers in GST.
      gst_conv_stride (int): Stride size of conv layers in GST.
      gst_gru_layers (int): The number of GRU layers in GST.
      gst_gru_units (int): The number of GRU units in GST.
      init_type (str): How to initialize transformer parameters.
      init_enc_alpha (float): Initial value of alpha in scaled pos encoding of the
          encoder.
      init_dec_alpha (float): Initial value of alpha in scaled pos encoding of the
          decoder.
      use_masking (bool): Whether to apply masking for padded part in loss
          calculation.
      use_weighted_masking (bool): Whether to apply weighted masking in loss
          calculation.
docstr_tok: ["Initialize","ProDiff","module",".","Args",":","idim","(","int",")",":","Dimension","of","the","inputs",".","odim","(","int",")",":","Dimension","of","the","outputs",".","elayers","(","int",")",":","Number","of","encoder","layers",".","eunits","(","int",")",":","Number","of","encoder","hidden","units",".","dlayers","(","int",")",":","Number","of","decoder","layers",".","dunits","(","int",")",":","Number","of","decoder","hidden","units",".","postnet_layers","(","int",")",":","Number","of","postnet","layers",".","postnet_chans","(","int",")",":","Number","of","postnet","channels",".","postnet_filts","(","int",")",":","Kernel","size","of","postnet",".","postnet_dropout_rate","(","float",")",":","Dropout","rate","in","postnet",".","use_scaled_pos_enc","(","bool",")",":","Whether","to","use","trainable","scaled","pos","encoding",".","use_batch_norm","(","bool",")",":","Whether","to","use","batch","normalization","in","encoder","prenet",".","encoder_normalize_before","(","bool",")",":","Whether","to","apply","layernorm","layer","before","encoder","block",".","decoder_normalize_before","(","bool",")",":","Whether","to","apply","layernorm","layer","before","decoder","block",".","encoder_concat_after","(","bool",")",":","Whether","to","concatenate","attention","layer","'s","input","and","output","in","encoder",".","decoder_concat_after","(","bool",")",":","Whether","to","concatenate","attention","layer","'s","input","and","output","in","decoder",".","reduction_factor","(","int",")",":","Reduction","factor",".","encoder_type","(","str",")",":","Encoder","type","(","``","transformer","''","or","``","conformer","''",")",".","decoder_type","(","str",")",":","Decoder","type","(","``","transformer","''","or","``","conformer","''",")",".","transformer_enc_dropout_rate","(","float",")",":","Dropout","rate","in","encoder","except","attention","and","positional","encoding",".","transformer_enc_positional_dropout_rate","(","float",")",":","Dropout","rate","after","encoder","positional","encoding",".","transformer_enc_attn_dropout_rate","(","float",")",":","Dropout","rate","in","encoder","self-attention","module",".","transformer_dec_dropout_rate","(","float",")",":","Dropout","rate","in","decoder","except","attention","&","positional","encoding",".","transformer_dec_positional_dropout_rate","(","float",")",":","Dropout","rate","after","decoder","positional","encoding",".","transformer_dec_attn_dropout_rate","(","float",")",":","Dropout","rate","in","decoder","self-attention","module",".","conformer_rel_pos_type","(","str",")",":","Relative","pos","encoding","type","in","conformer",".","conformer_pos_enc_layer_type","(","str",")",":","Pos","encoding","layer","type","in","conformer",".","conformer_self_attn_layer_type","(","str",")",":","Self-attention","layer","type","in","conformer","conformer_activation_type","(","str",")",":","Activation","function","type","in","conformer",".","use_macaron_style_in_conformer",":","Whether","to","use","macaron","style","FFN",".","use_cnn_in_conformer",":","Whether","to","use","CNN","in","conformer",".","zero_triu",":","Whether","to","use","zero","triu","in","relative","self-attention","module",".","conformer_enc_kernel_size",":","Kernel","size","of","encoder","conformer",".","conformer_dec_kernel_size",":","Kernel","size","of","decoder","conformer",".","duration_predictor_layers","(","int",")",":","Number","of","duration","predictor","layers",".","duration_predictor_chans","(","int",")",":","Number","of","duration","predictor","channels",".","duration_predictor_kernel_size","(","int",")",":","Kernel","size","of","duration","predictor",".","duration_predictor_dropout_rate","(","float",")",":","Dropout","rate","in","duration","predictor",".","pitch_predictor_layers","(","int",")",":","Number","of","pitch","predictor","layers",".","pitch_predictor_chans","(","int",")",":","Number","of","pitch","predictor","channels",".","pitch_predictor_kernel_size","(","int",")",":","Kernel","size","of","pitch","predictor",".","pitch_predictor_dropout_rate","(","float",")",":","Dropout","rate","in","pitch","predictor",".","pitch_embed_kernel_size","(","float",")",":","Kernel","size","of","pitch","embedding",".","pitch_embed_dropout_rate","(","float",")",":","Dropout","rate","for","pitch","embedding",".","stop_gradient_from_pitch_predictor",":","Whether","to","stop","gradient","from","pitch","predictor","to","encoder",".","energy_predictor_layers","(","int",")",":","Number","of","energy","predictor","layers",".","energy_predictor_chans","(","int",")",":","Number","of","energy","predictor","channels",".","energy_predictor_kernel_size","(","int",")",":","Kernel","size","of","energy","predictor",".","energy_predictor_dropout_rate","(","float",")",":","Dropout","rate","in","energy","predictor",".","energy_embed_kernel_size","(","float",")",":","Kernel","size","of","energy","embedding",".","energy_embed_dropout_rate","(","float",")",":","Dropout","rate","for","energy","embedding",".","stop_gradient_from_energy_predictor",":","Whether","to","stop","gradient","from","energy","predictor","to","encoder",".","spks","(","Optional","[","int","]",")",":","Number","of","speakers",".","If","set","to",">","1",",","assume","that","the","sids","will","be","provided","as","the","input","and","use","sid","embedding","layer",".","langs","(","Optional","[","int","]",")",":","Number","of","languages",".","If","set","to",">","1",",","assume","that","the","lids","will","be","provided","as","the","input","and","use","sid","embedding","layer",".","spk_embed_dim","(","Optional","[","int","]",")",":","Speaker","embedding","dimension",".","If","set","to",">","0",",","assume","that","spembs","will","be","provided","as","the","input",".","spk_embed_integration_type",":","How","to","integrate","speaker","embedding",".","use_gst","(","str",")",":","Whether","to","use","global","style","token",".","gst_tokens","(","int",")",":","The","number","of","GST","embeddings",".","gst_heads","(","int",")",":","The","number","of","heads","in","GST","multihead","attention",".","gst_conv_layers","(","int",")",":","The","number","of","conv","layers","in","GST",".","gst_conv_chans_list",":","(","Sequence","[","int","]",")",":","List","of","the","number","of","channels","of","conv","layers","in","GST",".","gst_conv_kernel_size","(","int",")",":","Kernel","size","of","conv","layers","in","GST",".","gst_conv_stride","(","int",")",":","Stride","size","of","conv","layers","in","GST",".","gst_gru_layers","(","int",")",":","The","number","of","GRU","layers","in","GST",".","gst_gru_units","(","int",")",":","The","number","of","GRU","units","in","GST",".","init_type","(","str",")",":","How","to","initialize","transformer","parameters",".","init_enc_alpha","(","float",")",":","Initial","value","of","alpha","in","scaled","pos","encoding","of","the","encoder",".","init_dec_alpha","(","float",")",":","Initial","value","of","alpha","in","scaled","pos","encoding","of","the","decoder",".","use_masking","(","bool",")",":","Whether","to","apply","masking","for","padded","part","in","loss","calculation",".","use_weighted_masking","(","bool",")",":","Whether","to","apply","weighted","masking","in","loss","calculation","."]
code: def __init__(
      self,
      # network structure related
      idim: int,
      odim: int,
      adim: int = 384,
      aheads: int = 4,
      elayers: int = 6,
      eunits: int = 1536,
      postnet_layers: int = 0,
      postnet_chans: int = 512,
      postnet_filts: int = 5,
      postnet_dropout_rate: float = 0.5,
      positionwise_layer_type: str = "conv1d",
      positionwise_conv_kernel_size: int = 1,
      use_scaled_pos_enc: bool = True,
      use_batch_norm: bool = True,
      encoder_normalize_before: bool = True,
      encoder_concat_after: bool = False,
      reduction_factor: int = 1,
      encoder_type: str = "transformer",
      decoder_type: str = "diffusion",
      transformer_enc_dropout_rate: float = 0.1,
      transformer_enc_positional_dropout_rate: float = 0.1,
      transformer_enc_attn_dropout_rate: float = 0.1,
      # Denoiser Decoder
      denoiser_layers: int = 20,
      denoiser_channels: int = 256,
      diffusion_steps: int = 1000,
      diffusion_timescale: int = 1,
      diffusion_beta: float = 40.0,
      diffusion_scheduler: str = "vpsde",
      diffusion_cycle_ln: int = 1,
      # only for conformer
      conformer_rel_pos_type: str = "legacy",
      conformer_pos_enc_layer_type: str = "rel_pos",
      conformer_self_attn_layer_type: str = "rel_selfattn",
      conformer_activation_type: str = "swish",
      use_macaron_style_in_conformer: bool = True,
      use_cnn_in_conformer: bool = True,
      zero_triu: bool = False,
      conformer_enc_kernel_size: int = 7,
      # duration predictor
      duration_predictor_layers: int = 2,
      duration_predictor_chans: int = 384,
      duration_predictor_kernel_size: int = 3,
      duration_predictor_dropout_rate: float = 0.1,
      # energy predictor
      energy_predictor_layers: int = 2,
      energy_predictor_chans: int = 384,
      energy_predictor_kernel_size: int = 3,
      energy_predictor_dropout: float = 0.5,
      energy_embed_kernel_size: int = 9,
      energy_embed_dropout: float = 0.5,
      stop_gradient_from_energy_predictor: bool = False,
      # pitch predictor
      pitch_predictor_layers: int = 2,
      pitch_predictor_chans: int = 384,
      pitch_predictor_kernel_size: int = 3,
      pitch_predictor_dropout: float = 0.5,
      pitch_embed_kernel_size: int = 9,
      pitch_embed_dropout: float = 0.5,
      stop_gradient_from_pitch_predictor: bool = False,
      # extra embedding related
      spks: Optional[int] = None,
      langs: Optional[int] = None,
      spk_embed_dim: Optional[int] = None,
      spk_embed_integration_type: str = "add",
      use_gst: bool = False,
      gst_tokens: int = 10,
      gst_heads: int = 4,
      gst_conv_layers: int = 6,
      gst_conv_chans_list: Sequence[int] = (32, 32, 64, 64, 128, 128),
      gst_conv_kernel_size: int = 3,
      gst_conv_stride: int = 2,
      gst_gru_layers: int = 1,
      gst_gru_units: int = 128,
      # training related
      init_type: str = "xavier_uniform",
      init_enc_alpha: float = 1.0,
      init_dec_alpha: float = 1.0,
      use_masking: bool = False,
      use_weighted_masking: bool = False,
  ):
      """Initialize ProDiff module.
  
      Args:
          idim (int): Dimension of the inputs.
          odim (int): Dimension of the outputs.
          elayers (int): Number of encoder layers.
          eunits (int): Number of encoder hidden units.
          dlayers (int): Number of decoder layers.
          dunits (int): Number of decoder hidden units.
          postnet_layers (int): Number of postnet layers.
          postnet_chans (int): Number of postnet channels.
          postnet_filts (int): Kernel size of postnet.
          postnet_dropout_rate (float): Dropout rate in postnet.
          use_scaled_pos_enc (bool): Whether to use trainable scaled pos encoding.
          use_batch_norm (bool): Whether to use batch normalization in encoder prenet.
          encoder_normalize_before (bool): Whether to apply layernorm layer before
              encoder block.
          decoder_normalize_before (bool): Whether to apply layernorm layer before
              decoder block.
          encoder_concat_after (bool): Whether to concatenate attention layer's input
              and output in encoder.
          decoder_concat_after (bool): Whether to concatenate attention layer's input
              and output in decoder.
          reduction_factor (int): Reduction factor.
          encoder_type (str): Encoder type ("transformer" or "conformer").
          decoder_type (str): Decoder type ("transformer" or "conformer").
          transformer_enc_dropout_rate (float): Dropout rate in encoder except
              attention and positional encoding.
          transformer_enc_positional_dropout_rate (float): Dropout rate after encoder
              positional encoding.
          transformer_enc_attn_dropout_rate (float): Dropout rate in encoder
              self-attention module.
          transformer_dec_dropout_rate (float): Dropout rate in decoder except
              attention & positional encoding.
          transformer_dec_positional_dropout_rate (float): Dropout rate after decoder
              positional encoding.
          transformer_dec_attn_dropout_rate (float): Dropout rate in decoder
              self-attention module.
          conformer_rel_pos_type (str): Relative pos encoding type in conformer.
          conformer_pos_enc_layer_type (str): Pos encoding layer type in conformer.
          conformer_self_attn_layer_type (str): Self-attention layer type in conformer
          conformer_activation_type (str): Activation function type in conformer.
          use_macaron_style_in_conformer: Whether to use macaron style FFN.
          use_cnn_in_conformer: Whether to use CNN in conformer.
          zero_triu: Whether to use zero triu in relative self-attention module.
          conformer_enc_kernel_size: Kernel size of encoder conformer.
          conformer_dec_kernel_size: Kernel size of decoder conformer.
          duration_predictor_layers (int): Number of duration predictor layers.
          duration_predictor_chans (int): Number of duration predictor channels.
          duration_predictor_kernel_size (int): Kernel size of duration predictor.
          duration_predictor_dropout_rate (float): Dropout rate in duration predictor.
          pitch_predictor_layers (int): Number of pitch predictor layers.
          pitch_predictor_chans (int): Number of pitch predictor channels.
          pitch_predictor_kernel_size (int): Kernel size of pitch predictor.
          pitch_predictor_dropout_rate (float): Dropout rate in pitch predictor.
          pitch_embed_kernel_size (float): Kernel size of pitch embedding.
          pitch_embed_dropout_rate (float): Dropout rate for pitch embedding.
          stop_gradient_from_pitch_predictor: Whether to stop gradient from pitch
              predictor to encoder.
          energy_predictor_layers (int): Number of energy predictor layers.
          energy_predictor_chans (int): Number of energy predictor channels.
          energy_predictor_kernel_size (int): Kernel size of energy predictor.
          energy_predictor_dropout_rate (float): Dropout rate in energy predictor.
          energy_embed_kernel_size (float): Kernel size of energy embedding.
          energy_embed_dropout_rate (float): Dropout rate for energy embedding.
          stop_gradient_from_energy_predictor: Whether to stop gradient from energy
              predictor to encoder.
          spks (Optional[int]): Number of speakers. If set to > 1, assume that the
              sids will be provided as the input and use sid embedding layer.
          langs (Optional[int]): Number of languages. If set to > 1, assume that the
              lids will be provided as the input and use sid embedding layer.
          spk_embed_dim (Optional[int]): Speaker embedding dimension. If set to > 0,
              assume that spembs will be provided as the input.
          spk_embed_integration_type: How to integrate speaker embedding.
          use_gst (str): Whether to use global style token.
          gst_tokens (int): The number of GST embeddings.
          gst_heads (int): The number of heads in GST multihead attention.
          gst_conv_layers (int): The number of conv layers in GST.
          gst_conv_chans_list: (Sequence[int]):
              List of the number of channels of conv layers in GST.
          gst_conv_kernel_size (int): Kernel size of conv layers in GST.
          gst_conv_stride (int): Stride size of conv layers in GST.
          gst_gru_layers (int): The number of GRU layers in GST.
          gst_gru_units (int): The number of GRU units in GST.
          init_type (str): How to initialize transformer parameters.
          init_enc_alpha (float): Initial value of alpha in scaled pos encoding of the
              encoder.
          init_dec_alpha (float): Initial value of alpha in scaled pos encoding of the
              decoder.
          use_masking (bool): Whether to apply masking for padded part in loss
              calculation.
          use_weighted_masking (bool): Whether to apply weighted masking in loss
              calculation.
  
      """
      assert check_argument_types()
      super().__init__()
  
      # store hyperparameters
      self.idim = idim
      self.odim = odim
      self.eos = idim - 1
      self.reduction_factor = reduction_factor
      self.encoder_type = encoder_type
      self.decoder_type = decoder_type
      self.stop_gradient_from_pitch_predictor = (
          stop_gradient_from_pitch_predictor
      )
      self.stop_gradient_from_energy_predictor = (
          stop_gradient_from_energy_predictor
      )
      self.use_scaled_pos_enc = use_scaled_pos_enc
      self.use_gst = use_gst
  
      # use idx 0 as padding idx
      self.padding_idx = 0
  
      # get positional encoding class
      pos_enc_class = (
          ScaledPositionalEncoding
          if self.use_scaled_pos_enc
          else PositionalEncoding
      )
  
      # check relative positional encoding compatibility
      if "conformer" in [encoder_type, decoder_type]:
          if conformer_rel_pos_type == "legacy":
              if conformer_pos_enc_layer_type == "rel_pos":
                  conformer_pos_enc_layer_type = "legacy_rel_pos"
                  logging.warning(
                      "Fallback to conformer_pos_enc_layer_type = 'legacy_rel_pos' "
                      "due to the compatibility. If you want to use the new one, "
                      "please use conformer_pos_enc_layer_type = 'latest'."
                  )
              if conformer_self_attn_layer_type == "rel_selfattn":
                  conformer_self_attn_layer_type = "legacy_rel_selfattn"
                  logging.warning(
                      "Fallback to "
                      "conformer_self_attn_layer_type = 'legacy_rel_selfattn' "
                      "due to the compatibility. If you want to use the new one, "
                      "please use conformer_pos_enc_layer_type = 'latest'."
                  )
          elif conformer_rel_pos_type == "latest":
              assert conformer_pos_enc_layer_type != "legacy_rel_pos"
              assert (
                  conformer_self_attn_layer_type
                  != "legacy_rel_selfattn"
              )
          else:
              raise ValueError(
                  f"Unknown rel_pos_type: {conformer_rel_pos_type}"
              )
  
      # define encoder
      encoder_input_layer = torch.nn.Embedding(
          num_embeddings=idim,
          embedding_dim=adim,
          padding_idx=self.padding_idx,
      )
      if encoder_type == "transformer":
          self.encoder = TransformerEncoder(
              idim=idim,
              attention_dim=adim,
              attention_heads=aheads,
              linear_units=eunits,
              num_blocks=elayers,
              input_layer=encoder_input_layer,
              dropout_rate=transformer_enc_dropout_rate,
              positional_dropout_rate=transformer_enc_positional_dropout_rate,
              attention_dropout_rate=transformer_enc_attn_dropout_rate,
              pos_enc_class=pos_enc_class,
              normalize_before=encoder_normalize_before,
              concat_after=encoder_concat_after,
              positionwise_layer_type=positionwise_layer_type,
              positionwise_conv_kernel_size=positionwise_conv_kernel_size,
          )
      elif encoder_type == "conformer":
          self.encoder = ConformerEncoder(
              idim=idim,
              attention_dim=adim,
              attention_heads=aheads,
              linear_units=eunits,
              num_blocks=elayers,
              input_layer=encoder_input_layer,
              dropout_rate=transformer_enc_dropout_rate,
              positional_dropout_rate=transformer_enc_positional_dropout_rate,
              attention_dropout_rate=transformer_enc_attn_dropout_rate,
              normalize_before=encoder_normalize_before,
              concat_after=encoder_concat_after,
              positionwise_layer_type=positionwise_layer_type,
              positionwise_conv_kernel_size=positionwise_conv_kernel_size,
              macaron_style=use_macaron_style_in_conformer,
              pos_enc_layer_type=conformer_pos_enc_layer_type,
              selfattention_layer_type=conformer_self_attn_layer_type,
              activation_type=conformer_activation_type,
              use_cnn_module=use_cnn_in_conformer,
              cnn_module_kernel=conformer_enc_kernel_size,
              zero_triu=zero_triu,
          )
      else:
          raise ValueError(f"{encoder_type} is not supported.")
  
      # define GST
      if self.use_gst:
          self.gst = StyleEncoder(
              idim=odim,  # the input is mel-spectrogram
              gst_tokens=gst_tokens,
              gst_token_dim=adim,
              gst_heads=gst_heads,
              conv_layers=gst_conv_layers,
              conv_chans_list=gst_conv_chans_list,
              conv_kernel_size=gst_conv_kernel_size,
              conv_stride=gst_conv_stride,
              gru_layers=gst_gru_layers,
              gru_units=gst_gru_units,
          )
  
      # define spk and lang embedding
      self.spks = None
      if spks is not None and spks > 1:
          self.spks = spks
          self.sid_emb = torch.nn.Embedding(spks, adim)
      self.langs = None
      if langs is not None and langs > 1:
          self.langs = langs
          self.lid_emb = torch.nn.Embedding(langs, adim)
  
      # define additional projection for speaker embedding
      self.spk_embed_dim = None
      if spk_embed_dim is not None and spk_embed_dim > 0:
          self.spk_embed_dim = spk_embed_dim
          self.spk_embed_integration_type = spk_embed_integration_type
      if self.spk_embed_dim is not None:
          if self.spk_embed_integration_type == "add":
              self.projection = torch.nn.Linear(
                  self.spk_embed_dim, adim
              )
          else:
              self.projection = torch.nn.Linear(
                  adim + self.spk_embed_dim, adim
              )
  
      # define duration predictor
      self.duration_predictor = DurationPredictor(
          idim=adim,
          n_layers=duration_predictor_layers,
          n_chans=duration_predictor_chans,
          kernel_size=duration_predictor_kernel_size,
          dropout_rate=duration_predictor_dropout_rate,
      )
  
      # define pitch predictor
      self.pitch_predictor = VariancePredictor(
          idim=adim,
          n_layers=pitch_predictor_layers,
          n_chans=pitch_predictor_chans,
          kernel_size=pitch_predictor_kernel_size,
          dropout_rate=pitch_predictor_dropout,
      )
      # NOTE(kan-bayashi): We use continuous pitch + FastPitch style avg
      self.pitch_embed = torch.nn.Sequential(
          torch.nn.Conv1d(
              in_channels=1,
              out_channels=adim,
              kernel_size=pitch_embed_kernel_size,
              padding=(pitch_embed_kernel_size - 1) // 2,
          ),
          torch.nn.Dropout(pitch_embed_dropout),
      )
  
      # define energy predictor
      self.energy_predictor = VariancePredictor(
          idim=adim,
          n_layers=energy_predictor_layers,
          n_chans=energy_predictor_chans,
          kernel_size=energy_predictor_kernel_size,
          dropout_rate=energy_predictor_dropout,
      )
      # NOTE(kan-bayashi): We use continuous enegy + FastPitch style avg
      self.energy_embed = torch.nn.Sequential(
          torch.nn.Conv1d(
              in_channels=1,
              out_channels=adim,
              kernel_size=energy_embed_kernel_size,
              padding=(energy_embed_kernel_size - 1) // 2,
          ),
          torch.nn.Dropout(energy_embed_dropout),
      )
  
      # define length regulator
      self.length_regulator = LengthRegulator()
  
      # define decoder
      if decoder_type == "diffusion":
          self.decoder = SpectogramDenoiser(
              odim,
              adim=adim,
              layers=denoiser_layers,
              channels=denoiser_channels,
              timesteps=diffusion_steps,
              timescale=diffusion_timescale,
              max_beta=diffusion_beta,
              scheduler=diffusion_scheduler,
              cycle_length=diffusion_cycle_ln,
          )
      else:
          raise NotImplementedError(decoder_type)
  
      # define final projection
      if decoder_type != "diffusion":
          self.feat_out = torch.nn.Linear(adim, odim * reduction_factor)
      if reduction_factor > 1:
          raise NotImplementedError()
  
      # define postnet
      self.postnet = (
          None
          if postnet_layers == 0
          else Postnet(
              idim=idim,
              odim=odim,
              n_layers=postnet_layers,
              n_chans=postnet_chans,
              n_filts=postnet_filts,
              use_batch_norm=use_batch_norm,
              dropout_rate=postnet_dropout_rate,
          )
      )
  
      # initialize parameters
      self._reset_parameters(
          init_type=init_type,
          init_enc_alpha=init_enc_alpha,
          init_dec_alpha=init_dec_alpha,
      )
  
      # define criterions
      self.criterion = ProDiffLoss(
          use_masking=use_masking,
          use_weighted_masking=use_weighted_masking,
      )
  
imports: import logging
  from typing import Dict, Optional, Sequence, Tuple
  import torch
  import torch.nn.functional
  from typeguard import check_argument_types
  from espnet2.torch_utils.device_funcs import force_gatherable
  from espnet2.torch_utils.initialize import initialize
  from espnet2.tts.abs_tts import AbsTTS
  from espnet2.tts.fastspeech2.variance_predictor import VariancePredictor
  from espnet2.tts.gst.style_encoder import StyleEncoder
  from espnet2.tts.prodiff.denoiser import SpectogramDenoiser
  from espnet2.tts.prodiff.loss import ProDiffLoss
  from espnet.nets.pytorch_backend.conformer.encoder import Encoder
  from espnet.nets.pytorch_backend.fastspeech.duration_predictor import DurationPredictor
  from espnet.nets.pytorch_backend.fastspeech.length_regulator import LengthRegulator
  from espnet.nets.pytorch_backend.nets_utils import make_non_pad_mask, make_pad_mask
  from espnet.nets.pytorch_backend.tacotron2.decoder import Postnet
  from espnet.nets.pytorch_backend.transformer.embedding import PositionalEncoding, ScaledPositionalEncoding
  from espnet.nets.pytorch_backend.transformer.encoder import Encoder
label_desc: Use image node_id 1 to create a new ProDiff object from inherited base classes: AbsTTS with example: obj = ProDiff(idim, odim, adim, aheads, elayers, eunits, postnet_layers, postnet_chans, postnet_filts, postnet_dropout_rate, positionwise_layer_type, positionwise_conv_kernel_size, use_scaled_pos_enc, use_batch_norm, encoder_normalize_before, encoder_concat_after, reduction_factor, encoder_type, decoder_type, transformer_enc_dropout_rate, transformer_enc_positional_dropout_rate, transformer_enc_attn_dropout_rate, denoiser_layers, denoiser_channels, diffusion_steps, diffusion_timescale, diffusion_beta, diffusion_scheduler, diffusion_cycle_ln, conformer_rel_pos_type, conformer_pos_enc_layer_type, conformer__attn_layer_type, conformer_activation_type, use_macaron_style_in_conformer, use_cnn_in_conformer, zero_triu, conformer_enc_kernel_size, duration_predictor_layers, duration_predictor_chans, duration_predictor_kernel_size, duration_predictor_dropout_rate, energy_predictor_layers, energy_predictor_chans, energy_predictor_kernel_size, energy_predictor_dropout, energy_embed_kernel_size, energy_embed_dropout, stop_gradient_from_energy_predictor, pitch_predictor_layers, pitch_predictor_chans, pitch_predictor_kernel_size, pitch_predictor_dropout, pitch_embed_kernel_size, pitch_embed_dropout, stop_gradient_from_pitch_predictor, spks, langs, spk_embed_dim, spk_embed_integration_type, use_gst, gst_tokens, gst_heads, gst_conv_layers, gst_conv_chans_list, gst_conv_kernel_size, gst_conv_stride, gst_gru_layers, gst_gru_units, init_type, init_enc_alpha, init_dec_alpha, use_masking, use_weighted_masking)