class_docstr: A `PretrainedTransformerTokenizer` uses a model from HuggingFace's
  `transformers` library to tokenize some input text.  This often means wordpieces
  (where `'AllenNLP is awesome'` might get split into `['Allen', '##NL', '##P', 'is',
  'awesome']`), but it could also use byte-pair encoding, or some other tokenization, depending
  on the pretrained model that you're using.
  
  We take a model name as an input parameter, which we will pass to
  `AutoTokenizer.from_pretrained`.
  
  We also add special tokens relative to the pretrained model and truncate the sequences.
  
  This tokenizer also indexes tokens and adds the indexes to the `Token` fields so that
  they can be picked up by `PretrainedTransformerIndexer`.
  
  Registered as a `Tokenizer` with name "pretrained_transformer".
  
  # Parameters
  
  model_name : `str`
      The name of the pretrained wordpiece tokenizer to use.
  add_special_tokens : `bool`, optional, (default=`True`)
      If set to `True`, the sequences will be encoded with the special tokens relative
      to their model.
  max_length : `int`, optional (default=`None`)
      If set to a number, will limit the total sequence returned so that it has a maximum length.
  tokenizer_kwargs: `Dict[str, Any]`, optional (default = `None`)
      Dictionary with
      [additional arguments](https://github.com/huggingface/transformers/blob/155c782a2ccd103cf63ad48a2becd7c76a7d2115/transformers/tokenization_utils.py#L691)
      for `AutoTokenizer.from_pretrained`.
  verification_tokens: `Tuple[str, str]`, optional (default = `None`)
      A pair of tokens having different token IDs. It's used for reverse-engineering special tokens.
code: def __init__(
      self,
      model_name: str,
      add_special_tokens: bool = True,
      max_length: Optional[int] = None,
      tokenizer_kwargs: Optional[Dict[str, Any]] = None,
      verification_tokens: Optional[Tuple[str, str]] = None,
  ) -> None:
      if tokenizer_kwargs is None:
          tokenizer_kwargs = {}
      else:
          tokenizer_kwargs = tokenizer_kwargs.copy()
      # Note: Just because we request a fast tokenizer doesn't mean we get one.
      tokenizer_kwargs.setdefault("use_fast", True)
  
      self._tokenizer_kwargs = tokenizer_kwargs
      self._model_name = model_name
  
      from allennlp.common import cached_transformers
  
      self.tokenizer = cached_transformers.get_tokenizer(
          self._model_name,
          add_special_tokens=False,
          **self._tokenizer_kwargs,
      )
  
      self._add_special_tokens = add_special_tokens
      self._max_length = max_length
  
      self._tokenizer_lowercases = self.tokenizer_lowercases(
          self.tokenizer
      )
  
      if verification_tokens is None:
          try:
              self._reverse_engineer_special_tokens(
                  "a", "b", model_name, tokenizer_kwargs
              )
          except AssertionError:
              # For most transformer models, "a" and "b" work just fine as dummy tokens.  For a few,
              # they don't, and so we use "1" and "2" instead.
              self._reverse_engineer_special_tokens(
                  "1", "2", model_name, tokenizer_kwargs
              )
      else:
          token_a, token_b = verification_tokens
          self._reverse_engineer_special_tokens(
              token_a, token_b, model_name, tokenizer_kwargs
          )
  
imports: import copy
  import dataclasses
  import logging
  from typing import Any, Dict, List, Optional, Tuple, Iterable
  from transformers import PreTrainedTokenizer
  from allennlp.common.util import sanitize_wordpiece
  from allennlp.data.tokenizers.token_class import Token
  from allennlp.data.tokenizers.tokenizer import Tokenizer
label_desc: Use image node_id 1 to create a new PretrainedTransformerTokenizer object from inherited base classes: Tokenizer with example: obj = PretrainedTransformerTokenizer(model_name, add_special_tokens, max_length, tokenizer_kwargs, verification_tokens)