Do AND, OR strings have special meaning in PLY?
Question: When using PLY (<http://www.dabeaz.com/ply/>) I've noticed what seems to be a
very strange problem: when I'm using tokens like `&` for conjunction, the
program below works, but when I use `AND` in the same place, PLY claims syntax
error.

Program:

    
    
    #!/usr/bin/env python
    # -*- coding: utf-8 -*-
    
    import sys
    import os
    
    from ply import lex
    
    import ply.yacc as yacc
    
    parser = None
    lexer = None
    
    
    def flatten_list(lst):
        flat = []
        for x in lst:
            if isinstance(x, list):
                flat.extend(flatten_list(x))
            else:
                flat.append(x)
        return flat
    
    
    ############## Tokenization ##############
    
    tokens = (
        'number',
        'lparen',
        'rparen',
        'textw',
        'titlew',
        'qword',
        'AND'
    )
    
    t_lparen = r'\('
    t_rparen = r'\)'
    t_textw = r'TEXTW:'
    t_titlew = r'TITLEW:'
    t_qword = r'\w+'
    t_AND = r'AND'
    
    
    def t_number(t):
        r'\d+'
        t.value = int(t.value)
        return t
    
    
    t_ignore = ' \t'
    
    
    def t_error(t):
        raise ValueError(
            'Illegal character "{}" at position {}, query text: {}'.format(t.value[0], t.lexpos, t.lexer.lexdata))
    
    
    lexer = lex.lex()
    
    
    
    ################# Parsing #################
    
    
    def p_querylist_boolop(p):
        """querylist : subquery AND subquery"""
        print >> sys.stderr, 'p_querylist', list(p)
        p[0] = []
        p[0].append(p[1])
        p[0].append(p[3])
    
    
    def p_subquery(p):
        """subquery : lparen querykw qwordseq rparen"""
        print >> sys.stderr, 'p_subquery', list(p)
        p[0] = flatten_list(p[3])
    
    
    def p_querykw(p):
        """querykw : textw
                    | titlew"""
        print >> sys.stderr, 'p_querykw', list(p)
        p[0] = p[1]
    
    
    def p_qwordseq(p):
        """qwordseq : qwordseq qword
                      | qwordseq number
                      | qword
                      | number"""
        print >> sys.stderr, 'p_qwordseq', list(p)
        if p[0]:
            p[0].extend(p[1:])
        else:
            p[0] = p[1:]
    
    
    
    def p_error(p):
        global parser
        if p:
            tok = parser.token()
            if tok:
                msg = 'Syntax error in input, token "{}" at position {}, query text: {}'.format(tok.value, tok.lexpos,
                                                                                                lexer.lexdata)
                raise ValueError(msg)
        msg = 'Syntax error at the end of input, query text: {}'.format(lexer.lexdata)
        raise ValueError(msg)
    
    
    parser = yacc.yacc()
    # parser = yacc.yacc(debug=0, write_tables=0)
    
    
    def parse_query(q):
        return parser.parse(q)
    
    
    if __name__ == '__main__':
        query_texts = ["""(TEXTW: one article) AND (TEXTW: two books)"""]
        for qt in query_texts:
            res = parse_query(qt)
            print '***', res
    

This produces:

    
    
    ValueError: Syntax error in input, token "(" at position 19, query text: ( TEXTW: abc ) AND ( TEXTW: aaa )
    

However, when I change the following to:

    
    
    t_AND = r'&'
    query_texts = ["""(TEXTW: one article) & (TEXTW: two books)"""]
    

..it works just fine:

    
    
    *** [['one', 'article'], ['two', 'books']]
    


Answer: Ply has a slightly eccentric approach to ordering token regular expressions,
in part because it depends on the underlying python regular expression
library. Tokens defined with functions, such as your `number` token, are
recognuzed in the order they appear, and unlike many lexical scanner
generators, Ply makes no attempt to perform a longest match. Tokens defined by
assignment -- all your other token types -- have lower priority than
functions, and are placed in order by decreasing length (of the regular
expression).

The Ply manual (section 4.3) strongly suggests not using variable assignment
style for keyword tokens such as `AND`, because the pattern `r'AND'`, for
example, will recognize the first three characters of, for example,
`ANDROGYNOUS`, which you would probably expect to be a variable. Instead, it
recommends using a function with a simple pattern to first recognize all
keywords and variables as simple words, and then use a dictionary to recognize
the specific keywords. Sample code and a less telegraphic explanation are in
the Ply manual (in the section I cited above).

