class_docstr: Tokenizer transform abstract class.
docstr: Tokenize a list of words.
docstr_tok: ["Tokenize","a","list","of","words","."]
code: def _tokenize(self, tokens, side="src", is_train=False):
      """Tokenize a list of words."""
      # This method embeds a custom logic to correctly handle certain placeholders
      # in case the tokenizer doesn't preserve them.
      sentence = " ".join(tokens).replace(DefaultTokens.SEP, "\n")
      # Locate the end-of-sentence placeholders.
      sent_list = sentence.split(DefaultTokens.EOS)
      # Tokenize each sentence separately.
      segmented = []
      for _sentence in sent_list:
          # Locate the mask-before placeholders
          # (to zero-out the prompt loss during LM finetuning).
          _sentence_chunks = _sentence.split(DefaultTokens.MASK_BEFORE)
          # Tokenize each chunk separately and insert the padding token.
          # between each sequence of tokens.
          _sentence_tokens = []
          for _chunk in _sentence_chunks:
              _sentence_tokens += self.tokenize_string(
                  _chunk, side, is_train
              ) + [DefaultTokens.PAD]
          # Re-insert the eos token.
          segmented += _sentence_tokens[:-1] + [DefaultTokens.EOS]
      return segmented[:-1]
  
imports: import re
  from onmt.utils.logging import logger
  from onmt.transforms import register_transform
  from .transform import Transform, ObservableStats
  from onmt.constants import DefaultTokens
label_desc: Use image node_id 7 for calling the TokenizerTransform obj's underlying member method code with example usage: obj._tokenize(tokens, side, is_train) and returns: segmented