class_docstr: The encoder for ASR model of Conformer.
  Based on this paper:
  'Conformer: Convolution-augmented Transformer for Speech Recognition' by Anmol Gulati et al.
  https://arxiv.org/abs/2005.08100
  
  Args:
      feat_in (int): the size of feature channels
      n_layers (int): number of layers of ConformerBlock
      d_model (int): the hidden size of the model
      feat_out (int): the size of the output features
          Defaults to -1 (means feat_out is d_model)
      subsampling (str): the method of subsampling, choices=['vggnet', 'striding', 'dw-striding', 'stacking', 'stacking_norm']
          Defaults to striding.
      subsampling_factor (int): the subsampling factor which should be power of 2
          Defaults to 4.
      subsampling_conv_chunking_factor(int): optionally, force chunk inputs (helpful for large inputs)
          Should be power of 2, 1 (auto-chunking, default), or -1 (no chunking)
      subsampling_conv_channels (int): the size of the convolutions in the subsampling module
          Defaults to -1 which would set it to d_model.
      reduction (str, Optional): the method of reduction, choices=['pooling', 'striding']. If no value
          is passed, then no reduction is performed and the models runs with the original 4x subsampling.
      reduction_position (int, Optional): the index of the layer to apply reduction. If -1, apply reduction
          at the end.
      reduction_factor (int): the reduction factor which should be either 1 or a power of 2
          Defaults to 1.
      ff_expansion_factor (int): the expansion factor in feed forward layers
          Defaults to 4.
      self_attention_model (str): type of the attention layer and positional encoding
          'rel_pos': relative positional embedding and Transformer-XL
          'rel_pos_local_attn': relative positional embedding and Transformer-XL with local attention using
              overlapping chunks. Attention context is determined by att_context_size parameter.
          'abs_pos': absolute positional embedding and Transformer
          Default is rel_pos.
      pos_emb_max_len (int): the maximum length of positional embeddings
          Defaults to 5000
      n_heads (int): number of heads in multi-headed attention layers
          Defaults to 4.
      att_context_size (List[Union[List[int],int]]): specifies the context sizes on each side. Each context size should be a list of two integers like [100,100].
          A list of context sizes like [[100,100],[100,50]] can also be passed. -1 means unlimited context.
          Defaults to [-1,-1]
      att_context_probs (List[float]): a list of probabilities of each one of the att_context_size when a list of them is passed. If not specified, uniform distribution is being used.
          Defaults to None
      att_context_style (str): 'regular' or 'chunked_limited'.
          Defaults to 'regular'
      xscaling (bool): enables scaling the inputs to the multi-headed attention layers by sqrt(d_model)
          Defaults to True.
      untie_biases (bool): whether to not share (untie) the bias weights between layers of Transformer-XL
          Defaults to True.
      conv_kernel_size (int): the size of the convolutions in the convolutional modules
          Defaults to 31.
      conv_norm_type (str): the type of the normalization in the convolutional modules
          Defaults to 'batch_norm'.
      conv_context_size (list): it can be"causal" or a list of two integers while conv_context_size[0]+conv_context_size[1]+1==conv_kernel_size.
          None means [(conv_kernel_size-1)//2, (conv_kernel_size-1)//2], and 'causal' means [(conv_kernel_size-1), 0].
          Defaults to None.
      conv_dual_mode (bool): specifies if convolution should be dual mode when dual_offline mode is being used. When enables, the left half of the convolution kernel would get masked in streaming cases.
          Defaults to False
      dropout (float): the dropout rate used in all layers except the attention layers
          Defaults to 0.1.
      dropout_pre_encoder (float): the dropout rate used before the encoder
          Defaults to 0.1.
      dropout_emb (float): the dropout rate used for the positional embeddings
          Defaults to 0.1.
      dropout_att (float): the dropout rate used for the attention layer
          Defaults to 0.0.
      stochastic_depth_drop_prob (float): if non-zero, will randomly drop
          layers during training. The higher this value, the more often layers
          are dropped. Defaults to 0.0.
      stochastic_depth_mode (str): can be either "linear" or "uniform". If
          set to "uniform", all layers have the same probability of drop. If
          set to "linear", the drop probability grows linearly from 0 for the
          first layer to the desired value for the final layer. Defaults to
          "linear".
      stochastic_depth_start_layer (int): starting layer for stochastic depth.
          All layers before this will never be dropped. Note that drop
          probability will be adjusted accordingly if mode is "linear" when
          start layer is > 1. Defaults to 1.
      global_tokens (int): number of tokens to be used for global attention.
          Only relevant if self_attention_model is 'rel_pos_local_attn'.
          Defaults to 0.
      global_tokens_spacing (int): how far apart the global tokens are
          Defaults to 1.
      global_attn_separate (bool): whether the q, k, v layers used for global tokens should be separate.
          Defaults to False.
code: def _create_masks(
      self,
      att_context_size,
      padding_length,
      max_audio_length,
      offset,
      device,
  ):
      if self.self_attention_model != "rel_pos_local_attn":
          att_mask = torch.ones(
              1,
              max_audio_length,
              max_audio_length,
              dtype=torch.bool,
              device=device,
          )
  
          if self.att_context_style == "regular":
              if att_context_size[0] >= 0:
                  att_mask = att_mask.triu(
                      diagonal=-att_context_size[0]
                  )
              if att_context_size[1] >= 0:
                  att_mask = att_mask.tril(diagonal=att_context_size[1])
          elif self.att_context_style == "chunked_limited":
              # When right context is unlimited, just the left side of the masking need to get updated
              if att_context_size[1] == -1:
                  if att_context_size[0] >= 0:
                      att_mask = att_mask.triu(
                          diagonal=-att_context_size[0]
                      )
              else:
                  chunk_size = att_context_size[1] + 1
                  # left_chunks_num specifies the number of chunks to be visible by each chunk on the left side
                  if att_context_size[0] >= 0:
                      left_chunks_num = (
                          att_context_size[0] // chunk_size
                      )
                  else:
                      left_chunks_num = 10000
  
                  chunk_idx = torch.arange(
                      0,
                      max_audio_length,
                      dtype=torch.int,
                      device=att_mask.device,
                  )
                  chunk_idx = torch.div(
                      chunk_idx, chunk_size, rounding_mode="trunc"
                  )
                  diff_chunks = chunk_idx.unsqueeze(
                      1
                  ) - chunk_idx.unsqueeze(0)
                  chunked_limited_mask = torch.logical_and(
                      torch.le(diff_chunks, left_chunks_num),
                      torch.ge(diff_chunks, 0),
                  )
                  att_mask = torch.logical_and(
                      att_mask, chunked_limited_mask.unsqueeze(0)
                  )
      else:
          att_mask = None
  
      # pad_mask is the masking to be used to ignore paddings
      pad_mask = torch.arange(
          0, max_audio_length, device=device
      ).expand(padding_length.size(0), -1) < padding_length.unsqueeze(
          -1
      )
  
      if offset is not None:
          pad_mask_off = torch.arange(
              0, max_audio_length, device=device
          ).expand(padding_length.size(0), -1) >= offset.unsqueeze(-1)
          pad_mask = pad_mask_off.logical_and(pad_mask)
  
      if att_mask is not None:
          # pad_mask_for_att_mask is the mask which helps to ignore paddings
          pad_mask_for_att_mask = pad_mask.unsqueeze(1).repeat(
              [1, max_audio_length, 1]
          )
          pad_mask_for_att_mask = torch.logical_and(
              pad_mask_for_att_mask,
              pad_mask_for_att_mask.transpose(1, 2),
          )
          # att_mask is the masking to be used by the MHA layers to ignore the tokens not supposed to be visible
          att_mask = att_mask[:, :max_audio_length, :max_audio_length]
          # paddings should also get ignored, so pad_mask_for_att_mask is used to ignore their corresponding scores
          att_mask = torch.logical_and(
              pad_mask_for_att_mask,
              att_mask.to(pad_mask_for_att_mask.device),
          )
          att_mask = ~att_mask
  
      pad_mask = ~pad_mask
      return pad_mask, att_mask
  
imports: import math
  import random
  from collections import OrderedDict
  from dataclasses import dataclass
  from typing import List, Optional, Set
  import torch
  import torch.distributed
  import torch.nn
  from omegaconf import DictConfig, ListConfig, open_dict
  from nemo.collections.asr.models.configs import CacheAwareStreamingConfig
  from nemo.collections.asr.parts.mixins.streaming import StreamingEncoder
  from nemo.collections.asr.parts.submodules.causal_convs import CausalConv1D
  from nemo.collections.asr.parts.submodules.conformer_modules import ConformerLayer
  from nemo.collections.asr.parts.submodules.multi_head_attention import LocalAttRelPositionalEncoding, MultiHeadAttention, PositionalEncoding, RelPositionalEncoding, RelPositionMultiHeadAttention, RelPositionMultiHeadAttentionLongformer
  from nemo.collections.asr.parts.submodules.subsampling import ConvSubsampling, StackingSubsampling, SubsamplingReductionModule
  from nemo.collections.asr.parts.utils import adapter_utils
  from nemo.collections.asr.parts.utils.regularization_utils import compute_stochastic_depth_drop_probs
  from nemo.core.classes.common import typecheck
  from nemo.core.classes.exportable import Exportable
  from nemo.core.classes.mixins import AccessMixin, adapter_mixins
  from nemo.core.classes.module import NeuralModule
  from nemo.core.neural_types import AcousticEncodedRepresentation, ChannelType, LengthsType, NeuralType, SpectrogramType
  from nemo.utils import logging
label_desc: Use image node_id 15 for calling the ConformerEncoder obj's underlying member method code with example usage: obj._create_masks(att_context_size, padding_length, max_audio_length, offset, device) and returns: pad_mask,  att_mask