code: def tokenize(self, text):
      tokens = []
      self.lexer.input(text)
      while True:
          tok = self.lexer.token()
          if not tok:
              break
          tokens.append(tok)
      return tokens
  
imports: from __future__ import generators
  import sys
  import re
  import copy
  import time
  import os.path
label_desc: Use image node_id 2 for calling the Preprocessor obj's underlying member method code with example usage: obj.tokenize(text) and returns: tokens