class_docstr: Base class for tokenizers.
code: def _convert_words_to_tokens(
      words: List[Text], text: Text
  ) -> List[Token]:
      running_offset = 0
      tokens = []
  
      for word in words:
          word_offset = text.index(word, running_offset)
          word_len = len(word)
          running_offset = word_offset + word_len
          tokens.append(Token(word, word_offset))
  
      return tokens
  
imports: import abc
  import logging
  import re
  from typing import Text, List, Dict, Any, Optional
  from rasa.engine.graph import ExecutionContext, GraphComponent
  from rasa.engine.storage.resource import Resource
  from rasa.engine.storage.storage import ModelStorage
  from rasa.shared.nlu.training_data.training_data import TrainingData
  from rasa.shared.nlu.training_data.message import Message
  from rasa.nlu.constants import TOKENS_NAMES, MESSAGE_ATTRIBUTES
  from rasa.shared.nlu.constants import INTENT, INTENT_RESPONSE_KEY, RESPONSE_IDENTIFIER_DELIMITER, ACTION_NAME
  import rasa.shared.utils.io
label_desc: Use image node_id 9 for calling the Tokenizer obj's underlying member method code with example usage: obj._convert_words_to_tokens(words, text) and returns: tokens