docstr: Tokenize a document with the PyThaiNLP tokenizer and wrap the results into a Doc object.
          
docstr_tok: ["Tokenize","a","document","with","the","PyThaiNLP","tokenizer","and","wrap","the","results","into","a","Doc","object","."]
code: def process(self, document):
      """Tokenize a document with the PyThaiNLP tokenizer and wrap the results into a Doc object."""
      if isinstance(document, doc.Document):
          text = document.text
      else:
          text = document
      if not isinstance(text, str):
          raise Exception(
              "Must supply a string or Stanza Document object to the PyThaiNLP tokenizer."
          )
  
      sentences = []
      current_sentence = []
      offset = 0
  
      if self.no_ssplit:
          # skip sentence segmentation
          sent_strs = [text]
      else:
          sent_strs = self.pythai_sent_tokenize(text, engine="crfcut")
      for sent_str in sent_strs:
          for token_str in self.pythai_word_tokenize(
              sent_str, engine="newmm"
          ):
              # by default pythainlp will output whitespace as a token
              # we need to skip these tokens to be consistent with other tokenizers
              if token_str.isspace():
                  offset += len(token_str)
                  continue
  
              # create token entry
              token_entry = {
                  doc.TEXT: token_str,
                  doc.MISC: f"{doc.START_CHAR}={offset}|{doc.END_CHAR}={offset+len(token_str)}",
              }
              current_sentence.append(token_entry)
              offset += len(token_str)
  
          # finish sentence
          sentences.append(current_sentence)
          current_sentence = []
  
      if len(current_sentence) > 0:
          sentences.append(current_sentence)
  
      return doc.Document(sentences, text)
  
imports: from stanza.models.common import doc
  from stanza.pipeline._constants import TOKENIZE
  from stanza.pipeline.processor import ProcessorVariant, register_processor_variant
label_desc: Use image node_id 2 for calling the PyThaiNLPTokenizer obj's underlying member method code with example usage: obj.process(document) and returns: doc