class_docstr: Simple text streamer that prints the token(s) to stdout as soon as entire words are formed.
  
  <Tip warning={true}>
  
  The API for the streamer classes is still under development and may change in the future.
  
  </Tip>
  
  Parameters:
      tokenizer (`AutoTokenizer`):
          The tokenized used to decode the tokens.
      skip_prompt (`bool`, *optional*, defaults to `False`):
          Whether to skip the prompt to `.generate()` or not. Useful e.g. for chatbots.
      decode_kwargs (`dict`, *optional*):
          Additional keyword arguments to pass to the tokenizer's `decode` method.
  
  Examples:
  
      ```python
      >>> from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
  
      >>> tok = AutoTokenizer.from_pretrained("gpt2")
      >>> model = AutoModelForCausalLM.from_pretrained("gpt2")
      >>> inputs = tok(["An increasing sequence: one,"], return_tensors="pt")
      >>> streamer = TextStreamer(tok)
  
      >>> # Despite returning the usual output, the streamer will also print the generated text to stdout.
      >>> _ = model.generate(**inputs, streamer=streamer, max_new_tokens=20)
      An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,
      ```
class_docstr_tok: ["Simple","text","streamer","that","prints","the","token","(","s",")","to","stdout","as","soon","as","entire","words","are","formed",".","<","Tip","warning=","{","true","}",">","The","API","for","the","streamer","classes","is","still","under","development","and","may","change","in","the","future",".","<","\/Tip",">","Parameters",":","tokenizer","(","`","AutoTokenizer","`",")",":","The","tokenized","used","to","decode","the","tokens",".","skip_prompt","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","Whether","to","skip","the","prompt","to","`",".generate","(",")","`","or","not",".","Useful","e.g",".","for","chatbots",".","decode_kwargs","(","`","dict","`",",","*","optional","*",")",":","Additional","keyword","arguments","to","pass","to","the","tokenizer","'s","`","decode","`","method",".","Examples",":","``","`","python",">",">",">","from","transformers","import","AutoModelForCausalLM",",","AutoTokenizer",",","TextStreamer",">",">",">","tok","=","AutoTokenizer.from_pretrained","(","``","gpt2","''",")",">",">",">","model","=","AutoModelForCausalLM.from_pretrained","(","``","gpt2","''",")",">",">",">","inputs","=","tok","(","[","``","An","increasing","sequence",":","one",",","''","]",",","return_tensors=","''","pt","''",")",">",">",">","streamer","=","TextStreamer","(","tok",")",">",">",">","#","Despite","returning","the","usual","output",",","the","streamer","will","also","print","the","generated","text","to","stdout",".",">",">",">","_","=","model.generate","(","*","*","inputs",",","streamer=streamer",",","max_new_tokens=20",")","An","increasing","sequence",":","one",",","two",",","three",",","four",",","five",",","six",",","seven",",","eight",",","nine",",","ten",",","eleven",",","``","`"]
docstr: Checks whether CP is the codepoint of a CJK character.
docstr_tok: ["Checks","whether","CP","is","the","codepoint","of","a","CJK","character","."]
code: def _is_chinese_char(self, cp):
      """Checks whether CP is the codepoint of a CJK character."""
      # This defines a "chinese character" as anything in the CJK Unicode block:
      #   https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block)
      #
      # Note that the CJK Unicode block is NOT all Japanese and Korean characters,
      # despite its name. The modern Korean Hangul alphabet is a different block,
      # as is Japanese Hiragana and Katakana. Those alphabets are used to write
      # space-separated words, so they are not treated specially and handled
      # like the all of the other languages.
      if (
          (cp >= 0x4E00 and cp <= 0x9FFF)
          or (cp >= 0x3400 and cp <= 0x4DBF)  #
          or (cp >= 0x20000 and cp <= 0x2A6DF)  #
          or (cp >= 0x2A700 and cp <= 0x2B73F)  #
          or (cp >= 0x2B740 and cp <= 0x2B81F)  #
          or (cp >= 0x2B820 and cp <= 0x2CEAF)  #
          or (cp >= 0xF900 and cp <= 0xFAFF)
          or (cp >= 0x2F800 and cp <= 0x2FA1F)  #
      ):  #
          return True
  
      return False
  
file_path: transformers/src/transformers/generation/streamers.py
imports: from queue import Queue
  from typing import TYPE_CHECKING, Optional
label_desc: Use image node_id 5 for calling the TextStreamer obj's underlying member method code with example usage: obj._is_chinese_char(cp) and returns: False, True