class_docstr: Runs basic tokenization (punctuation splitting, lower casing, etc.).
class_docstr_tok: ["Runs","basic","tokenization","(","punctuation","splitting",",","lower","casing",",","etc",".",")","."]
docstr: Checks whether CP is the codepoint of a CJK character.
code: def _is_chinese_char(self, cp):
      """Checks whether CP is the codepoint of a CJK character."""
      # This defines a "chinese character" as anything in the CJK Unicode block:
      #     https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block)
      #
      # Note that the CJK Unicode block is NOT all Japanese and Korean characters,
      # despite its name. The modern Korean Hangul alphabet is a different block,
      # as is Japanese Hiragana and Katakana. Those alphabets are used to write
      # space-separated words, so they are not treated specially and handled
      # like the all of the other languages.
      if (
          (cp >= 0x4E00 and cp <= 0x9FFF)
          or (cp >= 0x3400 and cp <= 0x4DBF)  #
          or (cp >= 0x20000 and cp <= 0x2A6DF)  #
          or (cp >= 0x2A700 and cp <= 0x2B73F)  #
          or (cp >= 0x2B740 and cp <= 0x2B81F)  #
          or (cp >= 0x2B820 and cp <= 0x2CEAF)  #
          or (cp >= 0xF900 and cp <= 0xFAFF)
          or (cp >= 0x2F800 and cp <= 0x2FA1F)  #
      ):  #
          return True
  
      return False
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  import collections
  import unicodedata
  import six
  import sentencepiece
label_desc: Use image node_id 6 for calling the BasicTokenizer obj's underlying member method code with example usage: obj._is_chinese_char(cp) and returns: False, True