Python Lex-Yacc(PLY): Not recognizing start of line or start of string
Question: I am very new to [PLY](http://www.dabeaz.com/ply/) and a bit more than a
beginner to Python. I am trying to play around with
[PLY-3.4](http://www.dabeaz.com/ply/ply-3.4.tar.gz) and python 2.7 to learn
it. Please see the code below. I am trying to create a token QTAG which is a
string made of zero of more whitespaces followed by 'Q' or 'q', followed by
'.' and a positive integer and one or more whitespaces. For example VALID
QTAGs are

    
    
    "Q.11 "
    "  Q.12 "
    "q.13     "
    '''
       Q.14 
    '''
    

INVALID ones are

    
    
    "asdf Q.15 "
    "Q.  15 "
    

Here is my code:

    
    
    import ply.lex as lex
    
    class LqbLexer:
         # List of token names.   This is always required
         tokens =  [
            'QTAG',
            'INT'
            ]
    
    
         # Regular expression rules for simple tokens
    
        def t_QTAG(self,t):
            r'^[ \t]*[Qq]\.[0-9]+\s+'
            t.value = int(t.value.strip()[2:])
            return t
    
        # A regular expression rule with some action code
        # Note addition of self parameter since we're in a class
        def t_INT(self,t):
        r'\d+'
        t.value = int(t.value)   
        return t
    
    
        # Define a rule so we can track line numbers
        def t_newline(self,t):
            r'\n+'
            print "Newline found"
            t.lexer.lineno += len(t.value)
    
        # A string containing ignored characters (spaces and tabs)
        t_ignore  = ' \t'
    
        # Error handling rule
        def t_error(self,t):
            print "Illegal character '%s'" % t.value[0]
            t.lexer.skip(1)
    
        # Build the lexer
        def build(self,**kwargs):
            self.lexer = lex.lex(debug=1,module=self, **kwargs)
    
        # Test its output
        def test(self,data):
            self.lexer.input(data)
            while True:
                 tok = self.lexer.token()
                 if not tok: break
                 print tok
    
    # test it
    q = LqbLexer()
    q.build()
    #VALID inputs
    q.test("Q.11 ")
    q.test("  Q.12 ")
    q.test("q.13     ")
    q.test('''
       Q.14 
    ''')
    # INVALID ones are
    q.test("asdf Q.15 ")
    q.test("Q.  15 ")
    

The output I get is as follows:

    
    
    LexToken(QTAG,11,1,0)
    Illegal character 'Q'
    Illegal character '.'
    LexToken(INT,12,1,4)
    LexToken(QTAG,13,1,0)
    Newline found
    Illegal character 'Q'
    Illegal character '.'
    LexToken(INT,14,2,6)
    Newline found
    Illegal character 'a'
    Illegal character 's'
    Illegal character 'd'
    Illegal character 'f'
    Illegal character 'Q'
    Illegal character '.'
    LexToken(INT,15,3,7)
    Illegal character 'Q'
    Illegal character '.'
    LexToken(INT,15,3,4)
    

Notice that only the first and third of the valid inputs are correctly
tokenized. I am not able to figure out why my other valid inputs are not being
tokenized properly. In the doc string for t_QTAG:

  1. Replacing `'^'` with `'\A'` did not work.
  2. I tried by removing `'^'` . Then all the valid inputs get tokenized, but then second invalid input also gets tokenized.

Any help is appreciated in advance!

Thanks

PS: I joined the google-group ply-hack and tried posting there, but I could
not post either directly in the forum or through email. I am not sure if the
group is active anymore. Prof. Beazley is not responding either. Any ideas?


Answer: Finally I found the answer myself. Posting it so that others may find it
useful.

As @Tadgh rightly pointed out `t_ignore = ' \t'` consumes the spaces and tabs
and hence I will not be able to match as per above regex for `t_QTAG` and the
consequence is that the second valid input is not tokenized. By reading the
PLY documentation carefully, I learned that if the ordering of the regex for
tokens is to be maintained then they have to be defined in a function rather
than strings as was done for `t_ignore`. If strings are used then PLY
automatically orders them by longest to shortest length and appends them
_after_ the functions. Here `t_ignore` is special, I guess, that it is somehow
executed before anything else. This part is not clearly documented. The work
around for this to define a function with a new token, eg, `t_SPACETAB`,
_after_ `t_QTAG` and just do not return anything. With this, all the _valid_
inputs are correctly tokenized now, except the one with triple quotes (the
multi-line string containing `"Q.14"`). Also, the invalid ones are, as per
specification, not tokenized.

Multi-line string problem: It turns out that internally PLY uses `re` module.
In that module, `^` is interpreted only at the beginning of a _string_ and NOT
beginning of _every line_ , by default. To change that behavior, I need to
turn on the multi-line flag, which can be done within the regex using `(?m)`.
So, to process all the valid and invalid strings in my test properly, the
correct regex is:

`r'(?m)^\s*[Qq]\.[0-9]+\s+'`

Here is the corrected code with some more tests added:

    
    
    import ply.lex as lex
    
    class LqbLexer:
        # List of token names.   This is always required
    
        tokens = [
            'QTAG',
            'INT',
            'SPACETAB'
            ]
    
    
        # Regular expression rules for simple tokens
    
        def t_QTAG(self,t):
            # corrected regex
            r'(?m)^\s*[Qq]\.[0-9]+\s+'
            t.value = int(t.value.strip()[2:])
            return t
    
        # A regular expression rule with some action code
        # Note addition of self parameter since we're in a class
        def t_INT(self,t):
            r'\d+'
            t.value = int(t.value)    
            return t
    
        # Define a rule so we can track line numbers
        def t_newline(self,t):
            r'\n+'
            print "Newline found"
            t.lexer.lineno += len(t.value)
    
        # A string containing ignored characters (spaces and tabs)
        # Instead of t_ignore  = ' \t'
        def t_SPACETAB(self,t):
            r'[ \t]+'
            print "Space(s) and/or tab(s)"
    
        # Error handling rule
        def t_error(self,t):
            print "Illegal character '%s'" % t.value[0]
            t.lexer.skip(1)
    
        # Build the lexer
        def build(self,**kwargs):
            self.lexer = lex.lex(debug=1,module=self, **kwargs)
    
        # Test its output
        def test(self,data):
            self.lexer.input(data)
            while True:
                 tok = self.lexer.token()
                 if not tok: break
                 print tok
    
    # test it
    q = LqbLexer()
    q.build()
    print "-============Testing some VALID inputs===========-"
    q.test("Q.11 ")
    q.test("  Q.12 ")
    q.test("q.13     ")
    q.test("""
    
    
       Q.14
    """)
    q.test("""
    
    qewr
    dhdhg
    dfhg
       Q.15 asda
    
    """)
    
    # INVALID ones are
    print "-============Testing some INVALID inputs===========-"
    q.test("asdf Q.16 ")
    q.test("Q.  17 ")
    

Here is the output:

    
    
    -============Testing some VALID inputs===========-
    LexToken(QTAG,11,1,0)
    LexToken(QTAG,12,1,0)
    LexToken(QTAG,13,1,0)
    LexToken(QTAG,14,1,0)
    Newline found
    Illegal character 'q'
    Illegal character 'e'
    Illegal character 'w'
    Illegal character 'r'
    Newline found
    Illegal character 'd'
    Illegal character 'h'
    Illegal character 'd'
    Illegal character 'h'
    Illegal character 'g'
    Newline found
    Illegal character 'd'
    Illegal character 'f'
    Illegal character 'h'
    Illegal character 'g'
    Newline found
    LexToken(QTAG,15,6,18)
    Illegal character 'a'
    Illegal character 's'
    Illegal character 'd'
    Illegal character 'a'
    Newline found
    -============Testing some INVALID inputs===========-
    Illegal character 'a'
    Illegal character 's'
    Illegal character 'd'
    Illegal character 'f'
    Space(s) and/or tab(s)
    Illegal character 'Q'
    Illegal character '.'
    LexToken(INT,16,8,7)
    Space(s) and/or tab(s)
    Illegal character 'Q'
    Illegal character '.'
    Space(s) and/or tab(s)
    LexToken(INT,17,8,4)
    Space(s) and/or tab(s)
    

