docstr: Construct a PyThaiNLP-based tokenizer.
  
  Note that we always uses the default tokenizer of PyThaiNLP for sentence and word segmentation.
  Currently this is a CRF model for sentence segmentation and a dictionary-based model (newmm) for word segmentation.
docstr_tok: ["Construct","a","PyThaiNLP-based","tokenizer",".","Note","that","we","always","uses","the","default","tokenizer","of","PyThaiNLP","for","sentence","and","word","segmentation",".","Currently","this","is","a","CRF","model","for","sentence","segmentation","and","a","dictionary-based","model","(","newmm",")","for","word","segmentation","."]
code: def __init__(self, config):
      """Construct a PyThaiNLP-based tokenizer.
  
      Note that we always uses the default tokenizer of PyThaiNLP for sentence and word segmentation.
      Currently this is a CRF model for sentence segmentation and a dictionary-based model (newmm) for word segmentation.
      """
      if config["lang"] != "th":
          raise Exception(
              "PyThaiNLP tokenizer is only allowed in Thai pipeline."
          )
  
      check_pythainlp()
      from pythainlp.tokenize import (
          sent_tokenize as pythai_sent_tokenize,
      )
      from pythainlp.tokenize import (
          word_tokenize as pythai_word_tokenize,
      )
  
      self.pythai_sent_tokenize = pythai_sent_tokenize
      self.pythai_word_tokenize = pythai_word_tokenize
      self.no_ssplit = config.get("no_ssplit", False)
  
imports: from stanza.models.common import doc
  from stanza.pipeline._constants import TOKENIZE
  from stanza.pipeline.processor import ProcessorVariant, register_processor_variant
label_desc: Use image node_id 1 to create a new PyThaiNLPTokenizer object from inherited base classes: ProcessorVariant with example: obj = PyThaiNLPTokenizer(config)