title: Implementing a customizable Lexer class in Python
body: <p>I want to implement a customizable and extendable lexer-class.</p>
  
  <p><strong>My idea</strong></p>
  
  <p>2 different types of <em>handlers</em>:</p>
  
  <ol>
  <li>Character-set handlers</li>
  <li>Function handlers</li>
  </ol>
  
  <p>When a character is read, it is pushed through all handlers being registered to the lexer. As soon as a handler matches and returns a valid <em>Token</em>, it is returned. When no handler matches the character, a <em>worst-case</em> function is called, and if that doesn't return a valid <em>Token</em>, the next character is read and it goes on recursively.</p>
  
  <p>A <em>character-set handler</em> consists of a string and a function taking three arguments. The string represents a set of characters that will cause to invoke the function when the currently parsed character is contained in that string. The function may return <em>None</em> when it is not able to create a token of the following characters. The function <em>must</em> push all characters back being read in order to create a <em>Token</em> if they could not be handled. Characters that were pushed back to the scanner will be returned on the next time <code>scanner.read_next_char()</code> is called, this enables to read the input' byte-stream like the function hasn't ever touched the scanner.</p>
  
  <p>A <em>function handler</em> is a function taking three arguments that returns a valid <em>Token</em> or <em>None</em> if it could not create a valid <em>Token</em> with the characters following the character that invoked the handler, just like the functions being associated with a <em>character-set</em> above.</p>
  
  <p>The <em>worst-case</em> function is called when no handler could create a valid <em>Token</em> from the current and it's following characters and follows the above described pattern.</p>
  
  <p>Every handler takes three arguments, the <em>lexer</em> itself, the <em>scanner</em> used by the lexer (in order to read following characters) and the current character that actually invoked the handler.</p>
  
  <p>The Lexer itself and the handlers are separated.</p>
  
  <pre><code>class LexingRules(object):
  
      def __init__(self):
          self.charset_handlers  = []
          self.function_handlers = []
          self.worst_case        = None
  
      # convenient methods for adding handlers
  
  class Lexer(object):
  
      def __init__(self, scanner, rules):
          self.scanner = scanner
          self.rules   = rules
          self.current_token = None
          self.read_next_token()
  
      def read_next_token(self):
          current_char = self.scanner.current_char
  
          for charset, handler in self.rules.charset_handlers:
              if current_char in charset:
                  token = handler(self, self.scanner, current_char)
                  if isinstance(token, Token):
                      self.current_token = token
                      return token
          for function in self.rules.function_handlers:
              token = function(self, self.scanner, current_char)
              if isinstance(token, Token):
                  self.current_token = token
                  return token
          if self.rules.worst_case:
              token = self.rules.worst_case(self, self.scanner, current_char)
              if isinstance(token, Token):
                  self.current_token = token
                  return token
  
          self.scanner.read()
          return self.read_next_token()
  </code></pre>
  
  <p><strong>A possible LexingRules subclass</strong></p>
  
  <pre><code>import string
  class MyLexingRules(LexingRules):
  
      def __init__(self):
          super(MyLexingRules, self).__init__()
          self.charset_handlers.append(string.ascii_letters, self.handle_letters)
  
      def handle_letters(self, lexer, scanner, char):
          tkn = ''
          while scanner.current_char in string.ascii_letters:
              tkn += scanner.current_char
              scanner.read()
          return Token(tkn, TOKEN_ID_IDENTIFIER)
  </code></pre>
  
  <p><strong>The Aim</strong></p>
  
  <p><em>Separation of basic Lexer-implementation and hard-coded Lexing operations.</em> </p>
  
  <p>I'm currently at the beginning of a project of my own. I want to create an extendable compiled/translated language. Syntax and statement-extension etc. should be possible to create.</p>
  
  <p><strong>The Question</strong></p>
  
  <p>What do you think about the design of the lexer, do you have something to complain about, or an idea of how to improve it, adding more functionality?</p>