class_docstr: Tokenizer transform abstract class.
docstr: Apply subword-based tokenenization to src & tgt.
docstr_tok: ["Apply","subword-based","tokenenization","to","src","&","tgt","."]
code: def apply(self, example, is_train=False, stats=None, **kwargs):
      """Apply subword-based tokenenization to src & tgt."""
      src_out = self._tokenize(example["src"], "src", is_train)
      if example["tgt"] is not None:
          tgt_out = self._tokenize(example["tgt"], "tgt", is_train)
          if stats is not None:
              n_words = len(example["src"]) + len(example["tgt"])
              n_subwords = len(src_out) + len(tgt_out)
              stats.update(SubwordStats(n_subwords, n_words))
      else:
          tgt_out = None
          if stats is not None:
              n_words = len(example["src"])
              n_subwords = len(src_out)
              stats.update(SubwordStats(n_subwords, n_words))
      example["src"], example["tgt"] = src_out, tgt_out
      return example
  
imports: import re
  from onmt.utils.logging import logger
  from onmt.transforms import register_transform
  from .transform import Transform, ObservableStats
  from onmt.constants import DefaultTokens
label_desc: Use image node_id 8 for calling the TokenizerTransform obj's underlying member method code with example usage: obj.apply(example, is_train, stats) and returns: example