class_docstr: This is an implementation of transformer model for ASR.
  
  The architecture is based on the paper "Attention Is All You Need":
  https://arxiv.org/pdf/1706.03762.pdf
  
  Arguments
  ----------
  tgt_vocab: int
      Size of vocabulary.
  input_size: int
      Input feature size.
  d_model : int, optional
      Embedding dimension size.
      (default=512).
  nhead : int, optional
      The number of heads in the multi-head attention models (default=8).
  num_encoder_layers : int, optional
      The number of sub-encoder-layers in the encoder (default=6).
  num_decoder_layers : int, optional
      The number of sub-decoder-layers in the decoder (default=6).
  dim_ffn : int, optional
      The dimension of the feedforward network model (default=2048).
  dropout : int, optional
      The dropout value (default=0.1).
  activation : torch.nn.Module, optional
      The activation function of FFN layers.
      Recommended: relu or gelu (default=relu).
  positional_encoding: str, optional
      Type of positional encoding used. e.g. 'fixed_abs_sine' for fixed absolute positional encodings.
  normalize_before: bool, optional
      Whether normalization should be applied before or after MHA or FFN in Transformer layers.
      Defaults to True as this was shown to lead to better performance and training stability.
  kernel_size: int, optional
      Kernel size in convolutional layers when Conformer is used.
  bias: bool, optional
      Whether to use bias in Conformer convolutional layers.
  encoder_module: str, optional
      Choose between Conformer and Transformer for the encoder. The decoder is fixed to be a Transformer.
  conformer_activation: torch.nn.Module, optional
      Activation module used after Conformer convolutional layers. E.g. Swish, ReLU etc. it has to be a torch Module.
  branchformer_activation: torch.nn.Module, optional
      Activation module used within the Branchformer Encoder. E.g. Swish, ReLU etc. it has to be a torch Module.
  attention_type: str, optional
      Type of attention layer used in all Transformer or Conformer layers.
      e.g. regularMHA or RelPosMHA.
  max_length: int, optional
      Max length for the target and source sequence in input.
      Used for positional encodings.
  causal: bool, optional
      Whether the encoder should be causal or not (the decoder is always causal).
      If causal the Conformer convolutional layer is causal.
  csgu_linear_units: int, optional
      Number of neurons in the hidden linear units of the CSGU Module.
      -> Branchformer
  gate_activation: torch.nn.Module, optional
      Activation function used at the gate of the CSGU module.
      -> Branchformer
  use_linear_after_conv: bool, optional
      If True, will apply a linear transformation of size input_size//2.
      -> Branchformer
  
  Example
  -------
  >>> src = torch.rand([8, 120, 512])
  >>> tgt = torch.randint(0, 720, [8, 120])
  >>> net = TransformerASR(
  ...     720, 512, 512, 8, 1, 1, 1024, activation=torch.nn.GELU
  ... )
  >>> enc_out, dec_out = net.forward(src, tgt)
  >>> enc_out.shape
  torch.Size([8, 120, 512])
  >>> dec_out.shape
  torch.Size([8, 120, 512])
class_docstr_tok: ["This","is","an","implementation","of","transformer","model","for","ASR",".","The","architecture","is","based","on","the","paper","``","Attention","Is","All","You","Need","''",":","https",":","\/\/arxiv.org\/pdf\/1706.03762.pdf","Arguments","--","--","--","--","--","tgt_vocab",":","int","Size","of","vocabulary",".","input_size",":","int","Input","feature","size",".","d_model",":","int",",","optional","Embedding","dimension","size",".","(","default=512",")",".","nhead",":","int",",","optional","The","number","of","heads","in","the","multi-head","attention","models","(","default=8",")",".","num_encoder_layers",":","int",",","optional","The","number","of","sub-encoder-layers","in","the","encoder","(","default=6",")",".","num_decoder_layers",":","int",",","optional","The","number","of","sub-decoder-layers","in","the","decoder","(","default=6",")",".","dim_ffn",":","int",",","optional","The","dimension","of","the","feedforward","network","model","(","default=2048",")",".","dropout",":","int",",","optional","The","dropout","value","(","default=0.1",")",".","activation",":","torch.nn.Module",",","optional","The","activation","function","of","FFN","layers",".","Recommended",":","relu","or","gelu","(","default=relu",")",".","positional_encoding",":","str",",","optional","Type","of","positional","encoding","used",".","e.g",".","'fixed_abs_sine","'","for","fixed","absolute","positional","encodings",".","normalize_before",":","bool",",","optional","Whether","normalization","should","be","applied","before","or","after","MHA","or","FFN","in","Transformer","layers",".","Defaults","to","True","as","this","was","shown","to","lead","to","better","performance","and","training","stability",".","kernel_size",":","int",",","optional","Kernel","size","in","convolutional","layers","when","Conformer","is","used",".","bias",":","bool",",","optional","Whether","to","use","bias","in","Conformer","convolutional","layers",".","encoder_module",":","str",",","optional","Choose","between","Conformer","and","Transformer","for","the","encoder",".","The","decoder","is","fixed","to","be","a","Transformer",".","conformer_activation",":","torch.nn.Module",",","optional","Activation","module","used","after","Conformer","convolutional","layers",".","E.g",".","Swish",",","ReLU","etc",".","it","has","to","be","a","torch","Module",".","branchformer_activation",":","torch.nn.Module",",","optional","Activation","module","used","within","the","Branchformer","Encoder",".","E.g",".","Swish",",","ReLU","etc",".","it","has","to","be","a","torch","Module",".","attention_type",":","str",",","optional","Type","of","attention","layer","used","in","all","Transformer","or","Conformer","layers",".","e.g",".","regularMHA","or","RelPosMHA",".","max_length",":","int",",","optional","Max","length","for","the","target","and","source","sequence","in","input",".","Used","for","positional","encodings",".","causal",":","bool",",","optional","Whether","the","encoder","should","be","causal","or","not","(","the","decoder","is","always","causal",")",".","If","causal","the","Conformer","convolutional","layer","is","causal",".","csgu_linear_units",":","int",",","optional","Number","of","neurons","in","the","hidden","linear","units","of","the","CSGU","Module",".","-",">","Branchformer","gate_activation",":","torch.nn.Module",",","optional","Activation","function","used","at","the","gate","of","the","CSGU","module",".","-",">","Branchformer","use_linear_after_conv",":","bool",",","optional","If","True",",","will","apply","a","linear","transformation","of","size","input_size\/\/2",".","-",">","Branchformer","Example","--","--","--","-",">",">",">","src","=","torch.rand","(","[","8",",","120",",","512","]",")",">",">",">","tgt","=","torch.randint","(","0",",","720",",","[","8",",","120","]",")",">",">",">","net","=","TransformerASR","(","...","720",",","512",",","512",",","8",",","1",",","1",",","1024",",","activation=torch.nn.GELU","...",")",">",">",">","enc_out",",","dec_out","=","net.forward","(","src",",","tgt",")",">",">",">","enc_out.shape","torch.Size","(","[","8",",","120",",","512","]",")",">",">",">","dec_out.shape","torch.Size","(","[","8",",","120",",","512","]",")"]
docstr: This method implements a decoding step for the transformer model.
  
  Arguments
  ---------
  tgt : torch.Tensor
      The sequence to the decoder.
  encoder_out : torch.Tensor
      Hidden output of the encoder.
  enc_len : torch.LongTensor
      The actual length of encoder states.
docstr_tok: ["This","method","implements","a","decoding","step","for","the","transformer","model",".","Arguments","--","--","--","--","-","tgt",":","torch.Tensor","The","sequence","to","the","decoder",".","encoder_out",":","torch.Tensor","Hidden","output","of","the","encoder",".","enc_len",":","torch.LongTensor","The","actual","length","of","encoder","states","."]
code: def decode(self, tgt, encoder_out, enc_len=None):
      """This method implements a decoding step for the transformer model.
  
      Arguments
      ---------
      tgt : torch.Tensor
          The sequence to the decoder.
      encoder_out : torch.Tensor
          Hidden output of the encoder.
      enc_len : torch.LongTensor
          The actual length of encoder states.
      """
      tgt_mask = get_lookahead_mask(tgt)
      src_key_padding_mask = None
      if enc_len is not None:
          src_key_padding_mask = (1 - length_to_mask(enc_len)).bool()
  
      tgt = self.custom_tgt_module(tgt)
      if self.attention_type == "RelPosMHAXL":
          tgt = tgt + self.positional_encoding_decoder(tgt)
          pos_embs_encoder = None  # self.positional_encoding(src)
          pos_embs_target = None
      elif (
          self.positional_encoding_type == "fixed_abs_sine"
          or self.attention_type == "hypermixing"
      ):
          tgt = tgt + self.positional_encoding(
              tgt
          )  # add the encodings here
          pos_embs_target = None
          pos_embs_encoder = None
  
      prediction, self_attns, multihead_attns = self.decoder(
          tgt,
          encoder_out,
          tgt_mask=tgt_mask,
          memory_key_padding_mask=src_key_padding_mask,
          pos_embs_tgt=pos_embs_target,
          pos_embs_src=pos_embs_encoder,
      )
      return prediction, multihead_attns[-1]
  
imports: from dataclasses import dataclass
  import torch
  from torch import nn
  from typing import Any, Optional
  from speechbrain.nnet.linear import Linear
  from speechbrain.nnet.containers import ModuleList
  from speechbrain.lobes.models.transformer.Transformer import TransformerInterface, get_lookahead_mask, get_key_padding_mask, NormalizedEmbedding
  from speechbrain.nnet.activations import Swish
  from speechbrain.dataio.dataio import length_to_mask
  from speechbrain.utils.dynamic_chunk_training import DynChunkTrainConfig
label_desc: Use image node_id 3 for calling the TransformerASR obj's underlying member method code with example usage: obj.decode(tgt, encoder_out, enc_len) and returns: prediction,  multihead_attns