class_docstr: Base class inherited by all tokenizer classes. This class
  handles downloading and loading pre-trained tokenizer and adding tokens to
  the vocabulary.
  
  Derived class can set up a few special tokens to be used in common scripts
  and internals: :attr:`bos_token`, :attr:`eos_token`, :attr:`unk_token`,
  :attr:`sep_token`, :attr:`pad_token`, :attr:`cls_token`,
  :attr:`mask_token`, and :attr:`additional_special_tokens`.
  
  We defined an :attr:`added_tokens_encoder` to add new tokens to the
  vocabulary without having to handle the specific vocabulary augmentation
  methods of the various underlying dictionary structures (`BPE`,
  `sentencepiece` ...).
class_docstr_tok: ["Base","class","inherited","by","all","tokenizer","classes",".","This","class","handles","downloading","and","loading","pre-trained","tokenizer","and","adding","tokens","to","the","vocabulary",".","Derived","class","can","set","up","a","few","special","tokens","to","be","used","in","common","scripts","and","internals",":",":","attr",":","`","bos_token","`",",",":","attr",":","`","eos_token","`",",",":","attr",":","`","unk_token","`",",",":","attr",":","`","sep_token","`",",",":","attr",":","`","pad_token","`",",",":","attr",":","`","cls_token","`",",",":","attr",":","`","mask_token","`",",","and",":","attr",":","`","additional_special_tokens","`",".","We","defined","an",":","attr",":","`","added_tokens_encoder","`","to","add","new","tokens","to","the","vocabulary","without","having","to","handle","the","specific","vocabulary","augmentation","methods","of","the","various","underlying","dictionary","structures","(","`","BPE","`",",","`","sentencepiece","`","...",")","."]
docstr: Maps a string to a sequence of tokens (string), using the
  tokenizer. Split in words for word-based vocabulary or sub-words for
  sub-word-based vocabularies (`BPE`/`SentencePiece`/`WordPiece`).
  This function also takes care of the added tokens.
  
  Args:
      text: A input string.
  
  Return:
      A list of tokens.
docstr_tok: ["Maps","a","string","to","a","sequence","of","tokens","(","string",")",",","using","the","tokenizer",".","Split","in","words","for","word-based","vocabulary","or","sub-words","for","sub-word-based","vocabularies","(","`","BPE","`","\/","`","SentencePiece","`","\/","`","WordPiece","`",")",".","This","function","also","takes","care","of","the","added","tokens",".","Args",":","text",":","A","input","string",".","Return",":","A","list","of","tokens","."]
code: def map_text_to_token(
      self, text: Optional[str], **kwargs
  ) -> List[str]:
      r"""Maps a string to a sequence of tokens (string), using the
      tokenizer. Split in words for word-based vocabulary or sub-words for
      sub-word-based vocabularies (`BPE`/`SentencePiece`/`WordPiece`).
      This function also takes care of the added tokens.
  
      Args:
          text: A input string.
  
      Return:
          A list of tokens.
      """
  
      def split_on_tokens(tok_list, string):
          if not string:
              return []
          if not tok_list:
              return self._map_text_to_token(string, **kwargs)
          tok = tok_list[0]
          split_text = string.split(tok)
          return sum(
              (
                  split_on_tokens(tok_list[1:], sub_text.strip())
                  + [tok]
                  for sub_text in split_text
              ),
              [],
          )[:-1]
  
      added_tokens = (
          list(self.added_tokens_encoder.keys())
          + self.all_special_tokens
      )
      tokenized_text = split_on_tokens(added_tokens, text)
      return tokenized_text
  
imports: from typing import Any, Dict, List, Optional, Tuple, overload
  import json
  import os
  import warnings
  from texar.tf.module_base import ModuleBase
label_desc: Use image node_id 9 for calling the TokenizerBase obj's underlying member method code with example usage: obj.map_text_to_token(text) and returns: tokenized_text, sum, list, self