class_docstr: Base class inherited by all tokenizer classes. This class
  handles downloading and loading pre-trained tokenizer and adding tokens to
  the vocabulary.
  
  Derived class can set up a few special tokens to be used in common scripts
  and internals: :attr:`bos_token`, :attr:`eos_token`, :attr:`unk_token`,
  :attr:`sep_token`, :attr:`pad_token`, :attr:`cls_token`,
  :attr:`mask_token`, and :attr:`additional_special_tokens`.
  
  We defined an :attr:`added_tokens_encoder` to add new tokens to the
  vocabulary without having to handle the specific vocabulary augmentation
  methods of the various underlying dictionary structures (`BPE`,
  `sentencepiece` ...).
class_docstr_tok: ["Base","class","inherited","by","all","tokenizer","classes",".","This","class","handles","downloading","and","loading","pre-trained","tokenizer","and","adding","tokens","to","the","vocabulary",".","Derived","class","can","set","up","a","few","special","tokens","to","be","used","in","common","scripts","and","internals",":",":","attr",":","`","bos_token","`",",",":","attr",":","`","eos_token","`",",",":","attr",":","`","unk_token","`",",",":","attr",":","`","sep_token","`",",",":","attr",":","`","pad_token","`",",",":","attr",":","`","cls_token","`",",",":","attr",":","`","mask_token","`",",","and",":","attr",":","`","additional_special_tokens","`",".","We","defined","an",":","attr",":","`","added_tokens_encoder","`","to","add","new","tokens","to","the","vocabulary","without","having","to","handle","the","specific","vocabulary","augmentation","methods","of","the","various","underlying","dictionary","structures","(","`","BPE","`",",","`","sentencepiece","`","...",")","."]
docstr: Clean up a list of simple English tokenization artifacts like
  spaces before punctuations and abbreviated forms.
docstr_tok: ["Clean","up","a","list","of","simple","English","tokenization","artifacts","like","spaces","before","punctuations","and","abbreviated","forms","."]
code: def clean_up_tokenization(out_string: str) -> str:
      r"""Clean up a list of simple English tokenization artifacts like
      spaces before punctuations and abbreviated forms.
      """
      out_string = (
          out_string.replace(" .", ".")
          .replace(" ?", "?")
          .replace(" !", "!")
          .replace(" ,", ",")
          .replace(" ' ", "'")
          .replace(" n't", "n't")
          .replace(" 'm", "'m")
          .replace(" do not", " don't")
          .replace(" 's", "'s")
          .replace(" 've", "'ve")
          .replace(" 're", "'re")
      )
      return out_string
  
imports: from typing import Any, Dict, List, Optional, Tuple, overload
  import json
  import os
  import warnings
  from texar.tf.module_base import ModuleBase
label_desc: Use image node_id 27 for calling the TokenizerBase obj's underlying member method code with example usage: obj.clean_up_tokenization(out_string) and returns: out_string