class_docstr: A `PretrainedTransformerTokenizer` uses a model from HuggingFace's
  `transformers` library to tokenize some input text.  This often means wordpieces
  (where `'AllenNLP is awesome'` might get split into `['Allen', '##NL', '##P', 'is',
  'awesome']`), but it could also use byte-pair encoding, or some other tokenization, depending
  on the pretrained model that you're using.
  
  We take a model name as an input parameter, which we will pass to
  `AutoTokenizer.from_pretrained`.
  
  We also add special tokens relative to the pretrained model and truncate the sequences.
  
  This tokenizer also indexes tokens and adds the indexes to the `Token` fields so that
  they can be picked up by `PretrainedTransformerIndexer`.
  
  Registered as a `Tokenizer` with name "pretrained_transformer".
  
  # Parameters
  
  model_name : `str`
      The name of the pretrained wordpiece tokenizer to use.
  add_special_tokens : `bool`, optional, (default=`True`)
      If set to `True`, the sequences will be encoded with the special tokens relative
      to their model.
  max_length : `int`, optional (default=`None`)
      If set to a number, will limit the total sequence returned so that it has a maximum length.
  tokenizer_kwargs: `Dict[str, Any]`, optional (default = `None`)
      Dictionary with
      [additional arguments](https://github.com/huggingface/transformers/blob/155c782a2ccd103cf63ad48a2becd7c76a7d2115/transformers/tokenization_utils.py#L691)
      for `AutoTokenizer.from_pretrained`.
  verification_tokens: `Tuple[str, str]`, optional (default = `None`)
      A pair of tokens having different token IDs. It's used for reverse-engineering special tokens.
code: def add_special_tokens(
      self, tokens1: List[Token], tokens2: Optional[List[Token]] = None
  ) -> List[Token]:
      def with_new_type_id(
          tokens: List[Token], type_id: int
      ) -> List[Token]:
          return [
              dataclasses.replace(t, type_id=type_id) for t in tokens
          ]
  
      # Make sure we don't change the input parameters
      tokens2 = copy.deepcopy(tokens2)
  
      if tokens2 is None:
          return (
              self.single_sequence_start_tokens
              + with_new_type_id(tokens1, self.single_sequence_token_type_id)  # type: ignore
              + self.single_sequence_end_tokens
          )
      else:
          return (
              self.sequence_pair_start_tokens
              + with_new_type_id(tokens1, self.sequence_pair_first_token_type_id)  # type: ignore
              + self.sequence_pair_mid_tokens
              + with_new_type_id(tokens2, self.sequence_pair_second_token_type_id)  # type: ignore
              + self.sequence_pair_end_tokens
          )
  
imports: import copy
  import dataclasses
  import logging
  from typing import Any, Dict, List, Optional, Tuple, Iterable
  from transformers import PreTrainedTokenizer
  from allennlp.common.util import sanitize_wordpiece
  from allennlp.data.tokenizers.token_class import Token
  from allennlp.data.tokenizers.tokenizer import Tokenizer
label_desc: Use image node_id 10 for calling the PretrainedTransformerTokenizer obj's underlying member method code with example usage: obj.add_special_tokens(tokens1, tokens2) and returns: unknown, unknown, unknown