class_docstr: Runs basic tokenization (punctuation splitting, lower casing, etc.).
docstr: Tokenizes a piece of text.
code: def tokenize(self, text):
      """Tokenizes a piece of text."""
      text = convert_to_unicode(text)
      text, idxs = self._clean_text(text)
  
      # This was added on November 1st, 2018 for the multilingual and Chinese
      # models. This is also applied to the English models now, but it doesn't
      # matter since the English models were not trained on any Chinese data
      # and generally don't have any Chinese data in them (there are Chinese
      # characters in the vocabulary because Wikipedia does have some Chinese
      # words in the English Wikipedia.).
      text, idxs = self._tokenize_chinese_chars(text, idxs)
  
      orig_tokens, idxs = whitespace_tokenize(text, idxs)
      split_tokens = []
      split_token_idxs = []
      for token, idx in zip(orig_tokens, idxs):
          if self.do_lower_case:
              token = token.lower()
              token = self._run_strip_accents(token)
          toks, idx_offsets = self._run_split_on_punc(token)
          split_tokens.extend(toks)
          split_token_idxs.extend(
              [[idx[0] + i for i in sub] for sub in idx_offsets if sub]
          )
  
      output_tokens = []
      output_idxs = []
      for tok, idxs in zip(split_tokens, split_token_idxs):
          if not (tok == "" or all(_is_whitespace(c) for c in tok)):
              output_tokens.append(tok)
              output_idxs.append(idxs)
      return output_tokens, output_idxs
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  import collections
  import re
  import unicodedata
  import six
  import numpy
  import tensorflow
label_desc: Use image node_id 2 for calling the BasicTokenizer obj's underlying member method code with example usage: obj.tokenize(text) and returns: output_tokens,  output_idxs