class_docstr: A generic evaluation-time tokenizer, which leverages built-in tokenizers
  in sacreBLEU (https://github.com/mjpost/sacrebleu). It additionally provides
  lowercasing, punctuation removal and character tokenization, which are
  applied after sacreBLEU tokenization.
  
  Args:
      tokenizer_type (str): the type of sacreBLEU tokenizer to apply.
      lowercase (bool): lowercase the text.
      punctuation_removal (bool): remove punctuation (based on unicode
      category) from text.
      character_tokenization (bool): tokenize the text to characters.
code: def tokenize(self, sent: str):
      tokenized = self.tokenizer(sent)
  
      if self.punctuation_removal:
          tokenized = self.remove_punctuation(tokenized)
  
      if self.character_tokenization:
          tokenized = self.SPACE.join(
              list(tokenized.replace(self.SPACE, self.SPACE_ESCAPE))
          )
  
      if self.lowercase:
          tokenized = tokenized.lower()
  
      return tokenized
  
imports: import unicodedata
  import sacrebleu
  from fairseq.dataclass import ChoiceEnum
label_desc: Use image node_id 3 for calling the EvaluationTokenizer obj's underlying member method code with example usage: obj.tokenize(sent) and returns: tokenized