class_docstr: ASR BPE Mixin class that sets up a Tokenizer via a config
  
  This mixin class adds the method `_setup_tokenizer(...)`, which can be used by ASR models
  which depend on subword tokenization.
  
  The setup_tokenizer method adds the following parameters to the class -
      -   tokenizer_cfg: The resolved config supplied to the tokenizer (with `dir` and `type` arguments).
      -   tokenizer_dir: The directory path to the tokenizer vocabulary + additional metadata.
      -   tokenizer_type: The type of the tokenizer. Currently supports `bpe` and `wpe`, as well as `agg`.
      -   vocab_path: Resolved path to the vocabulary text file.
  
  In addition to these variables, the method will also instantiate and preserve a tokenizer
  (subclass of TokenizerSpec) if successful, and assign it to self.tokenizer.
  
  The mixin also supports aggregate tokenizers, which consist of ordinary, monolingual tokenizers.
  If a conversion between a monolongual and an aggregate tokenizer (or vice versa) is detected,
  all registered artifacts will be cleaned up.
code: def _setup_tokenizer(self, tokenizer_cfg: DictConfig):
      tokenizer_type = tokenizer_cfg.get("type")
      if tokenizer_type is None:
          raise ValueError("`tokenizer.type` cannot be None")
      elif tokenizer_type.lower() == "agg":
          self._setup_aggregate_tokenizer(tokenizer_cfg)
      else:
          self._setup_monolingual_tokenizer(tokenizer_cfg)
  
imports: import os
  from abc import ABC, abstractmethod
  from typing import List
  import torch
  from omegaconf import DictConfig, OmegaConf, open_dict
  import nemo.collections.asr.models
  from nemo.collections.asr.parts.mixins.asr_adapter_mixins import ASRAdapterModelMixin
  from nemo.collections.asr.parts.mixins.streaming import StreamingEncoder
  from nemo.collections.asr.parts.utils import asr_module_utils
  from nemo.collections.asr.parts.utils.rnnt_utils import Hypothesis
  from nemo.collections.common import tokenizers
  from nemo.utils import logging
label_desc: Use image node_id 1 for calling the ASRBPEMixin obj's underlying member method code with example usage: obj._setup_tokenizer(tokenizer_cfg) without return types