class_docstr: Tokenizer transform abstract class.
docstr: Initialize necessary options for Tokenizer.
docstr_tok: ["Initialize","necessary","options","for","Tokenizer","."]
code: def __init__(self, opts):
      """Initialize necessary options for Tokenizer."""
      super().__init__(opts)
  
imports: import re
  from onmt.utils.logging import logger
  from onmt.transforms import register_transform
  from .transform import Transform, ObservableStats
  from onmt.constants import DefaultTokens
label_desc: Use image node_id 1 to create a new TokenizerTransform object from inherited base classes: Transform with example: obj = TokenizerTransform(opts)