class_docstr: This is an implementation of self-attention encoder layer.
  Arguments
  ----------
  d_ffn: int, optional
      The dimension of the feedforward network model hidden layer.
  nhead: int
      The number of heads in the multi-head attention models (default=8).
  d_model: int
      The number of expected features in the encoder/decoder inputs (default=512).
  kdim: int, optional
      Dimension of the key.
  vdim: int, optional
      Dimension of the value.
  dropout: int, optional
      The dropout value.
  activation: torch.nn.Module, optional
      The activation function for Feed-Forward Netowrk layer,
      e.g., relu or gelu or swish.
  normalize_before: bool, optional
      Whether normalization should be applied before or after MHA or FFN in Transformer layers.
      Defaults to True as this was shown to lead to better performance and training stability.
  attention_type: str, optional
      Type of attention layer used in all Transformer or Conformer layers.
      e.g. regularMHA or RelPosMHA.
  ffn_type: str
      type of ffn: regularFFN/1dcnn
  ffn_cnn_kernel_size_list: list of int
      kernel size of 2 1d-convs if ffn_type is 1dcnn
  causal: bool, optional
      Whether the encoder should be causal or not (the decoder is always causal).
      If causal the Conformer convolutional layer is causal.
  Example
  -------
  >>> import torch
  >>> x = torch.rand((8, 60, 512))
  >>> net = TransformerEncoderLayer(512, 8, d_model=512)
  >>> output = net(x)
  >>> output[0].shape
  torch.Size([8, 60, 512])
code: def __init__(
      self,
      d_ffn,
      nhead,
      d_model,
      kdim=None,
      vdim=None,
      dropout=0.0,
      activation=nn.ReLU,
      normalize_before=False,
      attention_type="regularMHA",
      ffn_type="regularFFN",
      ffn_cnn_kernel_size_list=[3, 3],
      causal=False,
  ):
      super().__init__()
  
      if attention_type == "regularMHA":
          self.self_att = sb.nnet.attention.MultiheadAttention(
              nhead=nhead,
              d_model=d_model,
              dropout=dropout,
              kdim=kdim,
              vdim=vdim,
          )
  
      elif attention_type == "RelPosMHAXL":
          self.self_att = sb.nnet.attention.RelPosMHAXL(
              d_model, nhead, dropout, mask_pos_future=causal
          )
      elif attention_type == "hypermixing":
          self.self_att = sb.lobes.models.transformer.HyperMixing(
              input_output_dim=d_model,
              hypernet_size=d_ffn,
              tied=False,
              num_heads=nhead,
              fix_tm_hidden_size=False,
          )
  
      if ffn_type == "regularFFN":
          self.pos_ffn = sb.nnet.attention.PositionalwiseFeedForward(
              d_ffn=d_ffn,
              input_size=d_model,
              dropout=dropout,
              activation=activation,
          )
      elif ffn_type == "1dcnn":
          self.pos_ffn = nn.Sequential(
              Conv1d(
                  in_channels=d_model,
                  out_channels=d_ffn,
                  kernel_size=ffn_cnn_kernel_size_list[0],
                  padding="causal" if causal else "same",
              ),
              nn.ReLU(),
              Conv1d(
                  in_channels=d_ffn,
                  out_channels=d_model,
                  kernel_size=ffn_cnn_kernel_size_list[1],
                  padding="causal" if causal else "same",
              ),
          )
  
      self.norm1 = sb.nnet.normalization.LayerNorm(d_model, eps=1e-6)
      self.norm2 = sb.nnet.normalization.LayerNorm(d_model, eps=1e-6)
      self.dropout1 = torch.nn.Dropout(dropout)
      self.dropout2 = torch.nn.Dropout(dropout)
  
      self.normalize_before = normalize_before
      self.pos_ffn_type = ffn_type
  
imports: import math
  import torch
  import torch.nn
  import speechbrain
  from typing import Optional
  import numpy
  from .Conformer import ConformerEncoder
  from .Branchformer import BranchformerEncoder
  from speechbrain.nnet.activations import Swish
  from speechbrain.nnet.attention import RelPosEncXL
  from speechbrain.nnet.CNN import Conv1d
label_desc: Use image node_id 1 to create a new TransformerEncoderLayer object from inherited base classes: nn with example: obj = TransformerEncoderLayer(d_ffn, nhead, d_model, kdim, vdim, dropout, activation, normalize_before, attention_type, ffn_type, ffn_cnn_kernel_size_list, causal)