title: pyparsing nestedExpr and nested parentheses
body: <p>I am working on a very simple "querying syntax" usable by people with reasonable technical skills (i.e., not coders per se, but able to touch on the subject)</p>
  
  <p>A typical example of what they would enter on a form is:</p>
  
  <pre><code>address like street
  AND
  vote =  True
  AND
  (
    (
      age&gt;=25
      AND
      gender = M
    )
    OR
    (
      age between [20,30]
      AND
      gender = F
    )
    OR
    (
      age &gt;= 70
      AND
      eyes != blue
    )
  )
  </code></pre>
  
  <p>With</p>
  
  <ol>
  <li>no quote required</li>
  <li>potentially infinite nesting of parentheses</li>
  <li>simple AND|OR linking</li>
  </ol>
  
  <p>I am using pyparsing (well, trying to anyway) and reaching something:</p>
  
  <pre><code>from pyparsing import *
  
  OPERATORS = [
      '&lt;',
      '&lt;=',
      '&gt;',
      '&gt;=',
      '=',
      '!=',
      'like'
      'regexp',
      'between'
  ]
  
  unicode_printables = u''.join(unichr(c) for c in xrange(65536)
                                if not unichr(c).isspace())
  
  # user_input is the text sent by the client form
  user_input = ' '.join(user_input.split())
  user_input = '(' + user_input + ')'
  
  AND = Keyword("AND").setName('AND')
  OR = Keyword("OR").setName('OR')
  
  FIELD = Word(alphanums).setName('FIELD')
  OPERATOR = oneOf(OPERATORS).setName('OPERATOR')
  VALUE = Word(unicode_printables).setName('VALUE')
  CRITERION = FIELD + OPERATOR + VALUE
  
  QUERY = Forward()
  NESTED_PARENTHESES = nestedExpr('(', ')')
  QUERY &lt;&lt; ( CRITERION | AND | OR | NESTED_PARENTHESES )
  
  RESULT = QUERY.parseString(user_input)
  RESULT.pprint()
  </code></pre>
  
  <p>The output is:</p>
  
  <pre><code>[['address',
    'like',
    'street',
    'AND',
    'vote',
    '=',
    'True',
    'AND',
    [['age&gt;=25', 'AND', 'gender', '=', 'M'],
     'OR',
     ['age', 'between', '[20,30]', 'AND', 'gender', '=', 'F'],
     'OR',
     ['age', '&gt;=', '70', 'AND', 'eyes', '!=', 'blue']]]]
  </code></pre>
  
  <p>Which I am only partially happy with - the main reason being that the desired final output would look like this:</p>
  
  <pre><code>[
    {
      "field" : "address",
      "operator" : "like",
      "value" : "street",
    },
    'AND',
    {
      "field" : "vote",
      "operator" : "=",
      "value" : True,
    },
    'AND',
    [
      [
        {
          "field" : "age",
          "operator" : "&gt;=",
          "value" : 25,
        },
        'AND'
        {
          "field" : "gender",
          "operator" : "=",
          "value" : "M",
        }
      ],
      'OR',
      [
        {
          "field" : "age",
          "operator" : "between",
          "value" : [20,30],
        },
        'AND'
        {
          "field" : "gender",
          "operator" : "=",
          "value" : "F",
        }
      ],
      'OR',
      [
        {
          "field" : "age",
          "operator" : "&gt;=",
          "value" : 70,
        },
        'AND'
        {
          "field" : "eyes",
          "operator" : "!=",
          "value" : "blue",
        }
      ],
    ]
  ]
  </code></pre>
  
  <p>Many thanks!</p>
  
  <p><strong>EDIT</strong></p>
  
  <p>After Paul's answer, this is what the code looks like. Obviously it works much more nicely :-)</p>
  
  <pre><code>unicode_printables = u''.join(unichr(c) for c in xrange(65536)
                                if not unichr(c).isspace())
  
  user_input = ' '.join(user_input.split())
  
  AND = oneOf(['AND', '&amp;'])
  OR = oneOf(['OR', '|'])
  FIELD = Word(alphanums)
  OPERATOR = oneOf(OPERATORS)
  VALUE = Word(unicode_printables)
  COMPARISON = FIELD + OPERATOR + VALUE
  
  QUERY = infixNotation(
      COMPARISON,
      [
          (AND, 2, opAssoc.LEFT,),
          (OR, 2, opAssoc.LEFT,),
      ]
  )
  
  class ComparisonExpr:
      def __init__(self, tokens):
          self.tokens = tokens
  
      def __str__(self):
          return "Comparison:('field': {!r}, 'operator': {!r}, 'value': {!r})".format(*self.tokens.asList())
  
  COMPARISON.addParseAction(ComparisonExpr)
  
  RESULT = QUERY.parseString(user_input).asList()
  print type(RESULT)
  from pprint import pprint
  pprint(RESULT)
  </code></pre>
  
  <p>The output is:</p>
  
  <pre><code>[
    [
      &lt;[snip]ComparisonExpr instance at 0x043D0918&gt;,
      'AND',
      &lt;[snip]ComparisonExpr instance at 0x043D0F08&gt;,
      'AND',
      [
        [
          &lt;[snip]ComparisonExpr instance at 0x043D3878&gt;,
          'AND',
          &lt;[snip]ComparisonExpr instance at 0x043D3170&gt;
        ],
        'OR',
        [
          [
            &lt;[snip]ComparisonExpr instance at 0x043D3030&gt;,
            'AND',
            &lt;[snip]ComparisonExpr instance at 0x043D3620&gt;
          ],
          'AND',
          [
            &lt;[snip]ComparisonExpr instance at 0x043D3210&gt;,
            'AND',
            &lt;[snip]ComparisonExpr instance at 0x043D34E0&gt;
          ]
        ]
      ]
    ]
  ]
  </code></pre>
  
  <p>Is there a way to return RESULT with dictionaries and not <code>ComparisonExpr</code> instances?</p>
  
  <p>EDIT2</p>
  
  <p>Came up with a naive and very specific solution, but which works for me so far:</p>
  
  <pre><code>[snip]
  class ComparisonExpr:
      def __init__(self, tokens):
          self.tokens = tokens
  
      def __str__(self):
          return "Comparison:('field': {!r}, 'operator': {!r}, 'value': {!r})".format(*self.tokens.asList())
  
      def asDict(self):
          return {
              "field": self.tokens.asList()[0],
              "operator": self.tokens.asList()[1],
              "value": self.tokens.asList()[2]
          }
  
  [snip]
  RESULT = QUERY.parseString(user_input).asList()[0]
  def convert(list):
      final = []
      for item in list:
          if item.__class__.__name__ == 'ComparisonExpr':
              final.append(item.asDict())
          elif item in ['AND', 'OR']:
              final.append(item)
          elif item.__class__.__name__ == 'list':
              final.append(convert(item))
          else:
              print 'ooops forgotten something maybe?'
  
      return final
  
  FINAL = convert(RESULT)
  pprint(FINAL)
  </code></pre>
  
  <p>Which outputs:</p>
  
  <pre><code>[{'field': 'address', 'operator': 'LIKE', 'value': 'street'},
     'AND',
     {'field': 'vote', 'operator': '=', 'value': 'true'},
     'AND',
     [[{'field': 'age', 'operator': '&gt;=', 'value': '25'},
       'AND',
       {'field': 'gender', 'operator': '=', 'value': 'M'}],
      'OR',
      [[{'field': 'age', 'operator': 'BETWEEN', 'value': '[20,30]'},
        'AND',
        {'field': 'gender', 'operator': '=', 'value': 'F'}],
       'AND',
       [{'field': 'age', 'operator': '&gt;=', 'value': '70'},
        'AND',
        {'field': 'eyes', 'operator': '!=', 'value': 'blue'}]]]]
  </code></pre>
  
  <p>Again thanks to Paul for pointing me if a right direction!</p>
  
  <p>The only thing unknown left is for me to turn <code>'true'</code> into <code>True</code> and <code>'[20,30]'</code> into <code>[20, 30]</code>.</p>