class_docstr: A `PretrainedTransformerTokenizer` uses a model from HuggingFace's
  `transformers` library to tokenize some input text.  This often means wordpieces
  (where `'AllenNLP is awesome'` might get split into `['Allen', '##NL', '##P', 'is',
  'awesome']`), but it could also use byte-pair encoding, or some other tokenization, depending
  on the pretrained model that you're using.
  
  We take a model name as an input parameter, which we will pass to
  `AutoTokenizer.from_pretrained`.
  
  We also add special tokens relative to the pretrained model and truncate the sequences.
  
  This tokenizer also indexes tokens and adds the indexes to the `Token` fields so that
  they can be picked up by `PretrainedTransformerIndexer`.
  
  Registered as a `Tokenizer` with name "pretrained_transformer".
  
  # Parameters
  
  model_name : `str`
      The name of the pretrained wordpiece tokenizer to use.
  add_special_tokens : `bool`, optional, (default=`True`)
      If set to `True`, the sequences will be encoded with the special tokens relative
      to their model.
  max_length : `int`, optional (default=`None`)
      If set to a number, will limit the total sequence returned so that it has a maximum length.
  tokenizer_kwargs: `Dict[str, Any]`, optional (default = `None`)
      Dictionary with
      [additional arguments](https://github.com/huggingface/transformers/blob/155c782a2ccd103cf63ad48a2becd7c76a7d2115/transformers/tokenization_utils.py#L691)
      for `AutoTokenizer.from_pretrained`.
  verification_tokens: `Tuple[str, str]`, optional (default = `None`)
      A pair of tokens having different token IDs. It's used for reverse-engineering special tokens.
docstr: The huggingface tokenizers produce tokens that may or may not be slices from the
  original text.  Differences arise from lowercasing, Unicode normalization, and other
  kinds of normalization, as well as special characters that are included to denote
  various situations, such as "##" in BERT for word pieces from the middle of a word, or
  "Ġ" in RoBERTa for the beginning of words not at the start of a sentence.
  
  This code attempts to calculate character offsets while being tolerant to these
  differences. It scans through the text and the tokens in parallel, trying to match up
  positions in both. If it gets out of sync, it backs off to not adding any token
  indices, and attempts to catch back up afterwards. This procedure is approximate.
  Don't rely on precise results, especially in non-English languages that are far more
  affected by Unicode normalization.
docstr_tok: ["The","huggingface","tokenizers","produce","tokens","that","may","or","may","not","be","slices","from","the","original","text",".","Differences","arise","from","lowercasing",",","Unicode","normalization",",","and","other","kinds","of","normalization",",","as","well","as","special","characters","that","are","included","to","denote","various","situations",",","such","as","``","#","#","''","in","BERT","for","word","pieces","from","the","middle","of","a","word",",","or","''","\u0120","''","in","RoBERTa","for","the","beginning","of","words","not","at","the","start","of","a","sentence",".","This","code","attempts","to","calculate","character","offsets","while","being","tolerant","to","these","differences",".","It","scans","through","the","text","and","the","tokens","in","parallel",",","trying","to","match","up","positions","in","both",".","If","it","gets","out","of","sync",",","it","backs","off","to","not","adding","any","token","indices",",","and","attempts","to","catch","back","up","afterwards",".","This","procedure","is","approximate",".","Do","n't","rely","on","precise","results",",","especially","in","non-English","languages","that","are","far","more","affected","by","Unicode","normalization","."]
code: def _estimate_character_indices(
      self, text: str, token_ids: List[int]
  ) -> List[Optional[Tuple[int, int]]]:
      """
      The huggingface tokenizers produce tokens that may or may not be slices from the
      original text.  Differences arise from lowercasing, Unicode normalization, and other
      kinds of normalization, as well as special characters that are included to denote
      various situations, such as "##" in BERT for word pieces from the middle of a word, or
      "Ġ" in RoBERTa for the beginning of words not at the start of a sentence.
  
      This code attempts to calculate character offsets while being tolerant to these
      differences. It scans through the text and the tokens in parallel, trying to match up
      positions in both. If it gets out of sync, it backs off to not adding any token
      indices, and attempts to catch back up afterwards. This procedure is approximate.
      Don't rely on precise results, especially in non-English languages that are far more
      affected by Unicode normalization.
      """
  
      token_texts = [
          sanitize_wordpiece(t)
          for t in self.tokenizer.convert_ids_to_tokens(token_ids)
      ]
      token_offsets: List[Optional[Tuple[int, int]]] = [None] * len(
          token_ids
      )
      if self._tokenizer_lowercases:
          text = text.lower()
          token_texts = [t.lower() for t in token_texts]
  
      min_allowed_skipped_whitespace = 3
      allowed_skipped_whitespace = min_allowed_skipped_whitespace
  
      text_index = 0
      token_index = 0
      while text_index < len(text) and token_index < len(token_ids):
          token_text = token_texts[token_index]
          token_start_index = text.find(token_text, text_index)
  
          # Did we not find it at all?
          if token_start_index < 0:
              token_index += 1
              # When we skip a token, we increase our tolerance, so we have a chance of catching back up.
              allowed_skipped_whitespace += (
                  1 + min_allowed_skipped_whitespace
              )
              continue
  
          # Did we jump too far?
          non_whitespace_chars_skipped = sum(
              1
              for c in text[text_index:token_start_index]
              if not c.isspace()
          )
          if non_whitespace_chars_skipped > allowed_skipped_whitespace:
              # Too many skipped characters. Something is wrong. Ignore this token.
              token_index += 1
              # When we skip a token, we increase our tolerance, so we have a chance of catching back up.
              allowed_skipped_whitespace += (
                  1 + min_allowed_skipped_whitespace
              )
              continue
          allowed_skipped_whitespace = min_allowed_skipped_whitespace
  
          token_offsets[token_index] = (
              token_start_index,
              token_start_index + len(token_text),
          )
          text_index = token_start_index + len(token_text)
          token_index += 1
      return token_offsets
  
imports: import copy
  import dataclasses
  import logging
  from typing import Any, Dict, List, Optional, Tuple, Iterable
  from transformers import PreTrainedTokenizer
  from allennlp.common.util import sanitize_wordpiece
  from allennlp.data.tokenizers.token_class import Token
  from allennlp.data.tokenizers.tokenizer import Tokenizer
label_desc: Use image node_id 5 for calling the PretrainedTransformerTokenizer obj's underlying member method code with example usage: obj._estimate_character_indices(text, token_ids) and returns: token_offsets