class_docstr: A `PretrainedTransformerTokenizer` uses a model from HuggingFace's
  `transformers` library to tokenize some input text.  This often means wordpieces
  (where `'AllenNLP is awesome'` might get split into `['Allen', '##NL', '##P', 'is',
  'awesome']`), but it could also use byte-pair encoding, or some other tokenization, depending
  on the pretrained model that you're using.
  
  We take a model name as an input parameter, which we will pass to
  `AutoTokenizer.from_pretrained`.
  
  We also add special tokens relative to the pretrained model and truncate the sequences.
  
  This tokenizer also indexes tokens and adds the indexes to the `Token` fields so that
  they can be picked up by `PretrainedTransformerIndexer`.
  
  Registered as a `Tokenizer` with name "pretrained_transformer".
  
  # Parameters
  
  model_name : `str`
      The name of the pretrained wordpiece tokenizer to use.
  add_special_tokens : `bool`, optional, (default=`True`)
      If set to `True`, the sequences will be encoded with the special tokens relative
      to their model.
  max_length : `int`, optional (default=`None`)
      If set to a number, will limit the total sequence returned so that it has a maximum length.
  tokenizer_kwargs: `Dict[str, Any]`, optional (default = `None`)
      Dictionary with
      [additional arguments](https://github.com/huggingface/transformers/blob/155c782a2ccd103cf63ad48a2becd7c76a7d2115/transformers/tokenization_utils.py#L691)
      for `AutoTokenizer.from_pretrained`.
  verification_tokens: `Tuple[str, str]`, optional (default = `None`)
      A pair of tokens having different token IDs. It's used for reverse-engineering special tokens.
docstr: Tokenizes each word into wordpieces separately and returns the wordpiece IDs.
  Also calculates offsets such that tokens[offsets[i][0]:offsets[i][1] + 1]
  corresponds to the original i-th token.
  
  This function inserts special tokens.
docstr_tok: ["Tokenizes","each","word","into","wordpieces","separately","and","returns","the","wordpiece","IDs",".","Also","calculates","offsets","such","that","tokens","[","offsets","[","i","]","[","0","]",":","offsets","[","i","]","[","1","]","+","1","]","corresponds","to","the","original","i-th","token",".","This","function","inserts","special","tokens","."]
code: def intra_word_tokenize(
      self, string_tokens: List[str]
  ) -> Tuple[List[Token], List[Optional[Tuple[int, int]]]]:
      """
      Tokenizes each word into wordpieces separately and returns the wordpiece IDs.
      Also calculates offsets such that tokens[offsets[i][0]:offsets[i][1] + 1]
      corresponds to the original i-th token.
  
      This function inserts special tokens.
      """
      tokens, offsets = self._intra_word_tokenize(string_tokens)
      tokens = self.add_special_tokens(tokens)
      offsets = self._increment_offsets(
          offsets, len(self.single_sequence_start_tokens)
      )
      return tokens, offsets
  
imports: import copy
  import dataclasses
  import logging
  from typing import Any, Dict, List, Optional, Tuple, Iterable
  from transformers import PreTrainedTokenizer
  from allennlp.common.util import sanitize_wordpiece
  from allennlp.data.tokenizers.token_class import Token
  from allennlp.data.tokenizers.tokenizer import Tokenizer
label_desc: Use image node_id 8 for calling the PretrainedTransformerTokenizer obj's underlying member method code with example usage: obj.intra_word_tokenize(string_tokens) and returns: tokens,  offsets