class_docstr: A `PretrainedTransformerTokenizer` uses a model from HuggingFace's
  `transformers` library to tokenize some input text.  This often means wordpieces
  (where `'AllenNLP is awesome'` might get split into `['Allen', '##NL', '##P', 'is',
  'awesome']`), but it could also use byte-pair encoding, or some other tokenization, depending
  on the pretrained model that you're using.
  
  We take a model name as an input parameter, which we will pass to
  `AutoTokenizer.from_pretrained`.
  
  We also add special tokens relative to the pretrained model and truncate the sequences.
  
  This tokenizer also indexes tokens and adds the indexes to the `Token` fields so that
  they can be picked up by `PretrainedTransformerIndexer`.
  
  Registered as a `Tokenizer` with name "pretrained_transformer".
  
  # Parameters
  
  model_name : `str`
      The name of the pretrained wordpiece tokenizer to use.
  add_special_tokens : `bool`, optional, (default=`True`)
      If set to `True`, the sequences will be encoded with the special tokens relative
      to their model.
  max_length : `int`, optional (default=`None`)
      If set to a number, will limit the total sequence returned so that it has a maximum length.
  tokenizer_kwargs: `Dict[str, Any]`, optional (default = `None`)
      Dictionary with
      [additional arguments](https://github.com/huggingface/transformers/blob/155c782a2ccd103cf63ad48a2becd7c76a7d2115/transformers/tokenization_utils.py#L691)
      for `AutoTokenizer.from_pretrained`.
  verification_tokens: `Tuple[str, str]`, optional (default = `None`)
      A pair of tokens having different token IDs. It's used for reverse-engineering special tokens.
code: def _intra_word_tokenize(
      self, string_tokens: List[str]
  ) -> Tuple[List[Token], List[Optional[Tuple[int, int]]]]:
      tokens: List[Token] = []
      offsets: List[Optional[Tuple[int, int]]] = []
      for token_string in string_tokens:
          wordpieces = self.tokenizer.encode_plus(
              token_string,
              add_special_tokens=False,
              return_tensors=None,
              return_offsets_mapping=False,
              return_attention_mask=False,
          )
          wp_ids = wordpieces["input_ids"]
  
          if len(wp_ids) > 0:
              offsets.append(
                  (len(tokens), len(tokens) + len(wp_ids) - 1)
              )
              tokens.extend(
                  Token(text=wp_text, text_id=wp_id)
                  for wp_id, wp_text in zip(
                      wp_ids,
                      self.tokenizer.convert_ids_to_tokens(wp_ids),
                  )
              )
          else:
              offsets.append(None)
      return tokens, offsets
  
imports: import copy
  import dataclasses
  import logging
  from typing import Any, Dict, List, Optional, Tuple, Iterable
  from transformers import PreTrainedTokenizer
  from allennlp.common.util import sanitize_wordpiece
  from allennlp.data.tokenizers.token_class import Token
  from allennlp.data.tokenizers.tokenizer import Tokenizer
label_desc: Use image node_id 6 for calling the PretrainedTransformerTokenizer obj's underlying member method code with example usage: obj._intra_word_tokenize(string_tokens) and returns: tokens,  offsets