class_docstr: A `PretrainedTransformerTokenizer` uses a model from HuggingFace's
  `transformers` library to tokenize some input text.  This often means wordpieces
  (where `'AllenNLP is awesome'` might get split into `['Allen', '##NL', '##P', 'is',
  'awesome']`), but it could also use byte-pair encoding, or some other tokenization, depending
  on the pretrained model that you're using.
  
  We take a model name as an input parameter, which we will pass to
  `AutoTokenizer.from_pretrained`.
  
  We also add special tokens relative to the pretrained model and truncate the sequences.
  
  This tokenizer also indexes tokens and adds the indexes to the `Token` fields so that
  they can be picked up by `PretrainedTransformerIndexer`.
  
  Registered as a `Tokenizer` with name "pretrained_transformer".
  
  # Parameters
  
  model_name : `str`
      The name of the pretrained wordpiece tokenizer to use.
  add_special_tokens : `bool`, optional, (default=`True`)
      If set to `True`, the sequences will be encoded with the special tokens relative
      to their model.
  max_length : `int`, optional (default=`None`)
      If set to a number, will limit the total sequence returned so that it has a maximum length.
  tokenizer_kwargs: `Dict[str, Any]`, optional (default = `None`)
      Dictionary with
      [additional arguments](https://github.com/huggingface/transformers/blob/155c782a2ccd103cf63ad48a2becd7c76a7d2115/transformers/tokenization_utils.py#L691)
      for `AutoTokenizer.from_pretrained`.
  verification_tokens: `Tuple[str, str]`, optional (default = `None`)
      A pair of tokens having different token IDs. It's used for reverse-engineering special tokens.
docstr: This method only handles a single sentence (or sequence) of text.
docstr_tok: ["This","method","only","handles","a","single","sentence","(","or","sequence",")","of","text","."]
code: def tokenize(self, text: str) -> List[Token]:
      """
      This method only handles a single sentence (or sequence) of text.
      """
      max_length = self._max_length
      if max_length is not None and not self._add_special_tokens:
          max_length += self.num_special_tokens_for_sequence()
  
      encoded_tokens = self.tokenizer.encode_plus(
          text=text,
          add_special_tokens=True,
          max_length=max_length,
          truncation=True if max_length is not None else False,
          return_tensors=None,
          return_offsets_mapping=self.tokenizer.is_fast,
          return_attention_mask=False,
          return_token_type_ids=True,
          return_special_tokens_mask=True,
      )
      # token_ids contains a final list with ids for both regular and special tokens
      token_ids, token_type_ids, special_tokens_mask, token_offsets = (
          encoded_tokens["input_ids"],
          encoded_tokens["token_type_ids"],
          encoded_tokens["special_tokens_mask"],
          encoded_tokens.get("offset_mapping"),
      )
  
      # If we don't have token offsets, try to calculate them ourselves.
      if token_offsets is None:
          token_offsets = self._estimate_character_indices(
              text, token_ids
          )
  
      tokens = []
      for token_id, token_type_id, special_token_mask, offsets in zip(
          token_ids, token_type_ids, special_tokens_mask, token_offsets
      ):
          # In `special_tokens_mask`, 1s indicate special tokens and 0s indicate regular tokens.
          # NOTE: in transformers v3.4.0 (and probably older versions) the docstring
          # for `encode_plus` was incorrect as it had the 0s and 1s reversed.
          # https://github.com/huggingface/transformers/pull/7949 fixed this.
          if not self._add_special_tokens and special_token_mask == 1:
              continue
  
          if offsets is None or offsets[0] >= offsets[1]:
              start = None
              end = None
          else:
              start, end = offsets
  
          tokens.append(
              Token(
                  text=self.tokenizer.convert_ids_to_tokens(
                      token_id, skip_special_tokens=False
                  ),
                  text_id=token_id,
                  type_id=token_type_id,
                  idx=start,
                  idx_end=end,
              )
          )
  
      return tokens
  
imports: import copy
  import dataclasses
  import logging
  from typing import Any, Dict, List, Optional, Tuple, Iterable
  from transformers import PreTrainedTokenizer
  from allennlp.common.util import sanitize_wordpiece
  from allennlp.data.tokenizers.token_class import Token
  from allennlp.data.tokenizers.tokenizer import Tokenizer
label_desc: Use image node_id 4 for calling the PretrainedTransformerTokenizer obj's underlying member method code with example usage: obj.tokenize(text) and returns: tokens