class_docstr: Construct a CANINE tokenizer (i.e. a character splitter). It turns text into a sequence of characters, and then
  converts each character into its Unicode code point.
  
  [`CanineTokenizer`] inherits from [`PreTrainedTokenizer`].
  
  Refer to superclass [`PreTrainedTokenizer`] for usage examples and documentation concerning parameters.
  
  Args:
      model_max_length (`int`, *optional*, defaults to 2048):
              The maximum sentence length the model accepts.
class_docstr_tok: ["Construct","a","CANINE","tokenizer","(","i.e",".","a","character","splitter",")",".","It","turns","text","into","a","sequence","of","characters",",","and","then","converts","each","character","into","its","Unicode","code","point",".","[","`","CanineTokenizer","`","]","inherits","from","[","`","PreTrainedTokenizer","`","]",".","Refer","to","superclass","[","`","PreTrainedTokenizer","`","]","for","usage","examples","and","documentation","concerning","parameters",".","Args",":","model_max_length","(","`","int","`",",","*","optional","*",",","defaults","to","2048",")",":","The","maximum","sentence","length","the","model","accepts","."]
docstr: Converts a Unicode code point (integer) in a token (str). In case it's a special code point, convert to
  human-readable format.
docstr_tok: ["Converts","a","Unicode","code","point","(","integer",")","in","a","token","(","str",")",".","In","case","it","'s","a","special","code","point",",","convert","to","human-readable","format","."]
code: def _convert_id_to_token(self, index: int) -> str:
      """
      Converts a Unicode code point (integer) in a token (str). In case it's a special code point, convert to
      human-readable format.
      """
      try:
          if index in SPECIAL_CODEPOINTS:
              return SPECIAL_CODEPOINTS[index]
          return chr(index)
      except TypeError:
          raise ValueError(f"invalid id: {index}")
  
file_path: H2O/h2o_flexgen/benchmark/third_party/transformers/src/transformers/models/canine/tokenization_canine.py
imports: from typing import Dict, List, Optional
  from ...tokenization_utils import AddedToken, PreTrainedTokenizer
  from ...utils import logging
label_desc: Use image node_id 5 for calling the CanineTokenizer obj's underlying member method code with example usage: obj._convert_id_to_token(index) and returns: chr, SPECIAL_CODEPOINTS