class_docstr: Customtokenizer which will tokenize the input text as words or phases and convert the words (str) to an index (int) using the vocab.
  If you would like tokens, please use `hub.BertTokenizer`.
docstr: Converts a string in a sequence of tokens (string), using the tokenizer.
  Text in chinese will be splitted in words using the Word Segmentor (Baidu_LAC) defaultly.
  If cut_function is set, it will be splitted in words using cut_function.
  Args:
      text (`string`): The sequence to be encoded.
  Returns:
      split_tokens (`list`): split
docstr_tok: ["Converts","a","string","in","a","sequence","of","tokens","(","string",")",",","using","the","tokenizer",".","Text","in","chinese","will","be","splitted","in","words","using","the","Word","Segmentor","(","Baidu_LAC",")","defaultly",".","If","cut_function","is","set",",","it","will","be","splitted","in","words","using","cut_function",".","Args",":","text","(","`","string","`",")",":","The","sequence","to","be","encoded",".","Returns",":","split_tokens","(","`","list","`",")",":","split"]
code: def tokenize(self, text: str):
      """
      Converts a string in a sequence of tokens (string), using the tokenizer.
      Text in chinese will be splitted in words using the Word Segmentor (Baidu_LAC) defaultly.
      If cut_function is set, it will be splitted in words using cut_function.
      Args:
          text (`string`): The sequence to be encoded.
      Returns:
          split_tokens (`list`): split
      """
      if self.tokenize_chinese_chars:
          splitted_tokens = self.cut_function(text=text)
      else:
          splitted_tokens = self.basic_tokenizer.tokenize(text=text)
      return splitted_tokens
  
imports: import collections
  import inspect
  import os
  from typing import Callable, List, Union
  import paddlehub
  from paddlehub.utils.log import logger
  from paddlehub.text.bert_tokenizer import BasicTokenizer
  from paddlehub.text.utils import load_vocab, whitespace_tokenize
label_desc: Use image node_id 9 for calling the CustomTokenizer obj's underlying member method code with example usage: obj.tokenize(text) and returns: splitted_tokens