class_docstr: Re-implementation of Tercom Tokenizer in Python 3.
  
  See src/ter/core/Normalizer.java in https://github.com/jhclark/tercom Note that Python doesn't support named Unicode
  blocks so the mapping for relevant blocks was taken from here: https://unicode-table.com/en/blocks/
  
  This implementation follows the implementation from
  https://github.com/mjpost/sacrebleu/blob/master/sacrebleu/tokenizers/tokenizer_ter.py.
docstr: Initialize the tokenizer.
  
  Args:
      normalize: An indication whether a general tokenization to be applied.
      no_punctuation: An indication whteher a punctuation to be removed from the sentences.
      lowercase: An indication whether to enable case-insensitivity.
      asian_support: An indication whether asian characters to be processed.
docstr_tok: ["Initialize","the","tokenizer",".","Args",":","normalize",":","An","indication","whether","a","general","tokenization","to","be","applied",".","no_punctuation",":","An","indication","whteher","a","punctuation","to","be","removed","from","the","sentences",".","lowercase",":","An","indication","whether","to","enable","case-insensitivity",".","asian_support",":","An","indication","whether","asian","characters","to","be","processed","."]
code: def __init__(
      self,
      normalize: bool = False,
      no_punctuation: bool = False,
      lowercase: bool = True,
      asian_support: bool = False,
  ) -> None:
      """Initialize the tokenizer.
  
      Args:
          normalize: An indication whether a general tokenization to be applied.
          no_punctuation: An indication whteher a punctuation to be removed from the sentences.
          lowercase: An indication whether to enable case-insensitivity.
          asian_support: An indication whether asian characters to be processed.
  
      """
      self.normalize = normalize
      self.no_punctuation = no_punctuation
      self.lowercase = lowercase
      self.asian_support = asian_support
  
imports: import re
  from functools import lru_cache
  from typing import Dict, Iterator, List, Optional, Sequence, Tuple, Type, Union
  from torch import Tensor, tensor
  from torchmetrics.functional.text.helper import _flip_trace, _LevenshteinEditDistance, _trace_to_alignment, _validate_inputs
label_desc: Use image node_id 1 to create a new _TercomTokenizer object  with example: obj = _TercomTokenizer(normalize, no_punctuation, lowercase, asian_support)