class_docstr: Runs Wordseg tokenziation.
class_docstr_tok: ["Runs","Wordseg","tokenziation","."]
code: def __init__(
      self,
      vocab_file,
      do_lower_case=True,
      unk_token="[UNK]",
      split_token="\1",
  ):
      self.vocab = load_vocab(vocab_file)
      self.inv_vocab = {v: k for k, v in self.vocab.items()}
      self.tokenizer = sp.SentencePieceProcessor()
      self.tokenizer.Load(vocab_file + ".model")
  
      self.do_lower_case = do_lower_case
      self.unk_token = unk_token
      self.split_token = split_token
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  import collections
  import unicodedata
  import six
  import sentencepiece
label_desc: Use image node_id 1 to create a new WordsegTokenizer object from inherited base classes: object with example: obj = WordsegTokenizer(vocab_file, do_lower_case, unk_token, split_token)