class_docstr: Simple version FMM(Forward Maximun Matching) word tokenizer. This tokenizer can only
  be used in topic model demo, but not in real business application scenarios.
  
  Notes: This tokenizer can only recognize the words in the corresponding vocab file.
class_docstr_tok: ["Simple","version","FMM","(","Forward","Maximun","Matching",")","word","tokenizer",".","This","tokenizer","can","only","be","used","in","topic","model","demo",",","but","not","in","real","business","application","scenarios",".","Notes",":","This","tokenizer","can","only","recognize","the","words","in","the","corresponding","vocab","file","."]
docstr: Tokenize the input string `text`, and return the tokenize result.
          
docstr_tok: ["Tokenize","the","input","string","`","text","`",",","and","return","the","tokenize","result","."]
code: def tokenize(self, text):
      """Tokenize the input string `text`, and return the tokenize result."""
      text_len = len(text)
      result = []
      i = 0
      while i < text_len:
          word = found_word = ""
          # Deal with English characters.
          if self.__is_eng_char(text[i]):
              for j in range(i, text_len + 1):
                  if j < text_len and self.__is_eng_char(text[j]):
                      word += self.__tolower(text[j])
                  else:
                      # Forward matching by character granularity.
                      if word in self.__vocab:
                          result.append(word)
                      i = j - 1
                      break
          else:
              for j in range(i, min(i + self.__max_word_len, text_len)):
                  word += text[j]
                  if word in self.__vocab:
                      found_word = word
              if len(found_word) > 0:
                  result.append(found_word)
                  i += len(found_word) - 1
          i += 1
      return result
  
file_path: PaddleHub/modules/text/language_model/lda_news/tokenizer.py
label_desc: Use image node_id 2 for calling the SimpleTokenizer obj's underlying member method code with example usage: obj.tokenize(text) and returns: result