class_docstr: The encoder for ASR model of Conformer.
  Based on this paper:
  'Conformer: Convolution-augmented Transformer for Speech Recognition' by Anmol Gulati et al.
  https://arxiv.org/abs/2005.08100
  
  Args:
      feat_in (int): the size of feature channels
      n_layers (int): number of layers of ConformerBlock
      d_model (int): the hidden size of the model
      feat_out (int): the size of the output features
          Defaults to -1 (means feat_out is d_model)
      subsampling (str): the method of subsampling, choices=['vggnet', 'striding', 'dw-striding', 'stacking', 'stacking_norm']
          Defaults to striding.
      subsampling_factor (int): the subsampling factor which should be power of 2
          Defaults to 4.
      subsampling_conv_chunking_factor(int): optionally, force chunk inputs (helpful for large inputs)
          Should be power of 2, 1 (auto-chunking, default), or -1 (no chunking)
      subsampling_conv_channels (int): the size of the convolutions in the subsampling module
          Defaults to -1 which would set it to d_model.
      reduction (str, Optional): the method of reduction, choices=['pooling', 'striding']. If no value
          is passed, then no reduction is performed and the models runs with the original 4x subsampling.
      reduction_position (int, Optional): the index of the layer to apply reduction. If -1, apply reduction
          at the end.
      reduction_factor (int): the reduction factor which should be either 1 or a power of 2
          Defaults to 1.
      ff_expansion_factor (int): the expansion factor in feed forward layers
          Defaults to 4.
      self_attention_model (str): type of the attention layer and positional encoding
          'rel_pos': relative positional embedding and Transformer-XL
          'rel_pos_local_attn': relative positional embedding and Transformer-XL with local attention using
              overlapping chunks. Attention context is determined by att_context_size parameter.
          'abs_pos': absolute positional embedding and Transformer
          Default is rel_pos.
      pos_emb_max_len (int): the maximum length of positional embeddings
          Defaults to 5000
      n_heads (int): number of heads in multi-headed attention layers
          Defaults to 4.
      att_context_size (List[Union[List[int],int]]): specifies the context sizes on each side. Each context size should be a list of two integers like [100,100].
          A list of context sizes like [[100,100],[100,50]] can also be passed. -1 means unlimited context.
          Defaults to [-1,-1]
      att_context_probs (List[float]): a list of probabilities of each one of the att_context_size when a list of them is passed. If not specified, uniform distribution is being used.
          Defaults to None
      att_context_style (str): 'regular' or 'chunked_limited'.
          Defaults to 'regular'
      xscaling (bool): enables scaling the inputs to the multi-headed attention layers by sqrt(d_model)
          Defaults to True.
      untie_biases (bool): whether to not share (untie) the bias weights between layers of Transformer-XL
          Defaults to True.
      conv_kernel_size (int): the size of the convolutions in the convolutional modules
          Defaults to 31.
      conv_norm_type (str): the type of the normalization in the convolutional modules
          Defaults to 'batch_norm'.
      conv_context_size (list): it can be"causal" or a list of two integers while conv_context_size[0]+conv_context_size[1]+1==conv_kernel_size.
          None means [(conv_kernel_size-1)//2, (conv_kernel_size-1)//2], and 'causal' means [(conv_kernel_size-1), 0].
          Defaults to None.
      conv_dual_mode (bool): specifies if convolution should be dual mode when dual_offline mode is being used. When enables, the left half of the convolution kernel would get masked in streaming cases.
          Defaults to False
      dropout (float): the dropout rate used in all layers except the attention layers
          Defaults to 0.1.
      dropout_pre_encoder (float): the dropout rate used before the encoder
          Defaults to 0.1.
      dropout_emb (float): the dropout rate used for the positional embeddings
          Defaults to 0.1.
      dropout_att (float): the dropout rate used for the attention layer
          Defaults to 0.0.
      stochastic_depth_drop_prob (float): if non-zero, will randomly drop
          layers during training. The higher this value, the more often layers
          are dropped. Defaults to 0.0.
      stochastic_depth_mode (str): can be either "linear" or "uniform". If
          set to "uniform", all layers have the same probability of drop. If
          set to "linear", the drop probability grows linearly from 0 for the
          first layer to the desired value for the final layer. Defaults to
          "linear".
      stochastic_depth_start_layer (int): starting layer for stochastic depth.
          All layers before this will never be dropped. Note that drop
          probability will be adjusted accordingly if mode is "linear" when
          start layer is > 1. Defaults to 1.
      global_tokens (int): number of tokens to be used for global attention.
          Only relevant if self_attention_model is 'rel_pos_local_attn'.
          Defaults to 0.
      global_tokens_spacing (int): how far apart the global tokens are
          Defaults to 1.
      global_attn_separate (bool): whether the q, k, v layers used for global tokens should be separate.
          Defaults to False.
code: def __init__(
      self,
      feat_in,
      n_layers,
      d_model,
      feat_out=-1,
      causal_downsampling=False,
      subsampling="striding",
      subsampling_factor=4,
      subsampling_conv_chunking_factor=1,
      subsampling_conv_channels=-1,
      reduction=None,
      reduction_position=None,
      reduction_factor=1,
      ff_expansion_factor=4,
      self_attention_model="rel_pos",
      n_heads=4,
      att_context_size=None,
      att_context_probs=None,
      att_context_style="regular",
      xscaling=True,
      untie_biases=True,
      pos_emb_max_len=5000,
      conv_kernel_size=31,
      conv_norm_type="batch_norm",
      conv_context_size=None,
      dropout=0.1,
      dropout_pre_encoder=0.1,
      dropout_emb=0.1,
      dropout_att=0.0,
      stochastic_depth_drop_prob: float = 0.0,
      stochastic_depth_mode: str = "linear",
      stochastic_depth_start_layer: int = 1,
      global_tokens: int = 0,
      global_tokens_spacing: int = 1,
      global_attn_separate: bool = False,
  ):
      super().__init__()
      d_ff = d_model * ff_expansion_factor
      self.d_model = d_model
      self.n_layers = n_layers
      self._feat_in = feat_in
      self.att_context_style = att_context_style
      self.subsampling_factor = subsampling_factor
      self.subsampling_conv_chunking_factor = (
          subsampling_conv_chunking_factor
      )
  
      self.self_attention_model = self_attention_model
      self.global_tokens = global_tokens
      self.global_attn_separate = global_attn_separate
      self.global_tokens_spacing = global_tokens_spacing
  
      # Setting up the att_context_size
      (
          self.att_context_size_all,
          self.att_context_size,
          self.att_context_probs,
          self.conv_context_size,
      ) = self._calc_context_sizes(
          att_context_style=att_context_style,
          att_context_size=att_context_size,
          att_context_probs=att_context_probs,
          conv_context_size=conv_context_size,
          conv_kernel_size=conv_kernel_size,
      )
  
      if xscaling:
          self.xscale = math.sqrt(d_model)
      else:
          self.xscale = None
  
      # Subsampling
      if subsampling_conv_channels == -1:
          subsampling_conv_channels = d_model
      if subsampling and subsampling_factor > 1:
          if subsampling in ["stacking", "stacking_norm"]:
              # stacking_norm has an extra layer norm after stacking comparing to stacking
              self.pre_encode = StackingSubsampling(
                  subsampling_factor=subsampling_factor,
                  feat_in=feat_in,
                  feat_out=d_model,
                  norm=True
                  if subsampling == "stacking_norm"
                  else False,
              )
          else:
              self.pre_encode = ConvSubsampling(
                  subsampling=subsampling,
                  subsampling_factor=subsampling_factor,
                  feat_in=feat_in,
                  feat_out=d_model,
                  conv_channels=subsampling_conv_channels,
                  subsampling_conv_chunking_factor=subsampling_conv_chunking_factor,
                  activation=nn.ReLU(True),
                  is_causal=causal_downsampling,
              )
      else:
          self.pre_encode = nn.Linear(feat_in, d_model)
  
      # Reduction
      if reduction and reduction_factor > 1:
          assert (
              reduction_position >= -1 and reduction_position < n_layers
          )
          self.reduction_subsampling = SubsamplingReductionModule(
              reduction=reduction,
              d_model=d_model,
              reduction_factor=reduction_factor,
          )
          self.reduction_position = reduction_position
      else:
          self.reduction_subsampling = None
          self.reduction_position = None
  
      self._feat_out = d_model
  
      # Biases for relative positional encoding
      if not untie_biases and self_attention_model == "rel_pos":
          d_head = d_model // n_heads
          pos_bias_u = nn.Parameter(torch.Tensor(n_heads, d_head))
          pos_bias_v = nn.Parameter(torch.Tensor(n_heads, d_head))
          nn.init.zeros_(pos_bias_u)
          nn.init.zeros_(pos_bias_v)
      else:
          pos_bias_u = None
          pos_bias_v = None
  
      # Positional encodings
      self.pos_emb_max_len = pos_emb_max_len
      if self_attention_model == "rel_pos":
          self.pos_enc = RelPositionalEncoding(
              d_model=d_model,
              dropout_rate=dropout_pre_encoder,
              max_len=pos_emb_max_len,
              xscale=self.xscale,
              dropout_rate_emb=dropout_emb,
          )
      elif self_attention_model == "rel_pos_local_attn":
          if max(att_context_size) <= 0:
              raise ValueError(
                  "When using local attention, context size must be set > 0"
              )
          self.pos_enc = LocalAttRelPositionalEncoding(
              att_context_size=att_context_size,
              d_model=d_model,
              dropout_rate=dropout,
              max_len=pos_emb_max_len,
              xscale=self.xscale,
              dropout_rate_emb=dropout_emb,
          )
      elif self_attention_model == "abs_pos":
          pos_bias_u = None
          pos_bias_v = None
          self.pos_enc = PositionalEncoding(
              d_model=d_model,
              dropout_rate=dropout_pre_encoder,
              max_len=pos_emb_max_len,
              xscale=self.xscale,
          )
      else:
          raise ValueError(
              f"Not valid self_attention_model: '{self_attention_model}'!"
          )
  
      self.layers = nn.ModuleList()
      for i in range(n_layers):
          layer = ConformerLayer(
              d_model=d_model,
              d_ff=d_ff,
              self_attention_model=self_attention_model,
              global_tokens=global_tokens,
              global_tokens_spacing=global_tokens_spacing,
              global_attn_separate=global_attn_separate,
              n_heads=n_heads,
              conv_kernel_size=conv_kernel_size,
              conv_norm_type=conv_norm_type,
              conv_context_size=self.conv_context_size,
              dropout=dropout,
              dropout_att=dropout_att,
              pos_bias_u=pos_bias_u,
              pos_bias_v=pos_bias_v,
              att_context_size=self.att_context_size,
          )
          self.layers.append(layer)
  
      if feat_out > 0 and feat_out != self._feat_out:
          self.out_proj = nn.Linear(self._feat_out, feat_out)
          self._feat_out = feat_out
      else:
          self.out_proj = None
          self._feat_out = d_model
      self.set_max_audio_length(self.pos_emb_max_len)
      self.use_pad_mask = True
  
      self.setup_streaming_params()
      self.export_cache_support = False
  
      self.layer_drop_probs = compute_stochastic_depth_drop_probs(
          len(self.layers),
          stochastic_depth_drop_prob,
          stochastic_depth_mode,
          stochastic_depth_start_layer,
      )
      # will be set in self.forward() if defined in AccessMixin config
      self.interctc_capture_at_layers = None
  
imports: import math
  import random
  from collections import OrderedDict
  from dataclasses import dataclass
  from typing import List, Optional, Set
  import torch
  import torch.distributed
  import torch.nn
  from omegaconf import DictConfig, ListConfig, open_dict
  from nemo.collections.asr.models.configs import CacheAwareStreamingConfig
  from nemo.collections.asr.parts.mixins.streaming import StreamingEncoder
  from nemo.collections.asr.parts.submodules.causal_convs import CausalConv1D
  from nemo.collections.asr.parts.submodules.conformer_modules import ConformerLayer
  from nemo.collections.asr.parts.submodules.multi_head_attention import LocalAttRelPositionalEncoding, MultiHeadAttention, PositionalEncoding, RelPositionalEncoding, RelPositionMultiHeadAttention, RelPositionMultiHeadAttentionLongformer
  from nemo.collections.asr.parts.submodules.subsampling import ConvSubsampling, StackingSubsampling, SubsamplingReductionModule
  from nemo.collections.asr.parts.utils import adapter_utils
  from nemo.collections.asr.parts.utils.regularization_utils import compute_stochastic_depth_drop_probs
  from nemo.core.classes.common import typecheck
  from nemo.core.classes.exportable import Exportable
  from nemo.core.classes.mixins import AccessMixin, adapter_mixins
  from nemo.core.classes.module import NeuralModule
  from nemo.core.neural_types import AcousticEncodedRepresentation, ChannelType, LengthsType, NeuralType, SpectrogramType
  from nemo.utils import logging
label_desc: Use image node_id 8 to create a new ConformerEncoder object from inherited base classes: NeuralModule, StreamingEncoder, Exportable, AccessMixin with example: obj = ConformerEncoder(feat_in, n_layers, d_model, feat_out, causal_downsampling, subsampling, subsampling_factor, subsampling_conv_chunking_factor, subsampling_conv_channels, reduction, reduction_position, reduction_factor, ff_expansion_factor, _attention_model, n_heads, att_context_size, att_context_probs, att_context_style, xscaling, untie_biases, pos_emb_max_len, conv_kernel_size, conv_norm_type, conv_context_size, dropout, dropout_pre_encoder, dropout_emb, dropout_att, stochastic_depth_drop_prob, stochastic_depth_mode, stochastic_depth_start_layer, global_tokens, global_tokens_spacing, global_attn_separate)