extracting a parenthesized Python expression from a string
Question: I've been wondering about how hard it would be to write some Python code to
search a string for the index of a substring of the form `${`_expr_`}`, for
example, where _expr_ is meant to be a Python expression or something
resembling one. Given such a thing, one could easily imagine going on to check
the expression's syntax with `compile()`, evaluate it against a particular
scope with `eval()`, and perhaps even substitute the result into the original
string. People must do very similar things all the time.

I could imagine solving such a problem using a third-party parser generator
[oof], or by hand-coding some sort of state machine [eek], or perhaps by
convincing Python's own parser to do the heavy lifting somehow [hmm]. Maybe
there's a third-party templating library somewhere that can be made to do
exactly this. Maybe restricting the syntax of _expr_ is likely to be a
worthwhile compromise in terms of simplicity or execution time or cutting down
on external dependencies -- for example, maybe all I really need is something
that matches any _expr_ that has balanced curly braces.

What's your sense?

## Update:

Thanks very much for your responses so far! Looking back at what I wrote
yesterday, I'm not sure I was sufficiently clear about what I'm asking.
Template substitution is indeed an interesting problem, and probably much more
useful to many more people than the expression extraction subproblem I'm
wondering about, but I brought it up only as a simple example of how the
answer to my question might be useful in real life. Some other potential
applications might include passing the extracted expressions to a syntax
highlighter; passing the result to a real Python parser and looking at or
monkeying with the parse tree; or using the sequence of extracted expressions
to build up a larger Python program, perhaps in conjunction with some
information taken from the surrounding text.

The `${`_expr_`}` syntax I mentioned is also intended as an example, and in
fact I wonder if I shouldn't have used `$(`_expr_`)` as my example instead,
because it makes the potential drawbacks of the obvious approach, along the
lines of `re.finditer(r'$\{([^}]+)\}', s)`, a bit easier to see. Python
expressions can (and often do) contain the `)` (or `}`) character. It seems
possible that handling any of those cases might be much more trouble than it's
worth, but I'm not convinced of that yet. Please feel free to try to make this
case!

Prior to posting this question, I spent quite a bit of time looking at Python
template engines hoping that one might expose the sort of low-level
functionality I'm asking about -- namely, something that can find expressions
in a variety of contexts and tell me where they are rather than being limited
to finding expressions embedded using a single hard-coded syntax, always
evaluating them, and always substituting the results back into the original
string. I haven't figured out how to use any of them to solve my problem yet,
but I do very much appreciate the suggestions regarding more to look at (can't
believe I missed that wonderful list on the wiki!). The API documentation for
these things tends to be pretty high-level, and I'm not too familiar with the
internals of any of them, so I'm sure I could use help looking at those and
figuring out how to get them to do this sort of thing.

Thanks for your patience!


Answer: I think what you're asking about is being able to insert Python code into text
files to be evaluated. There are several modules that already exist to provide
this kind of functionality. You can check the Python.org [**Templating wiki
page**](http://wiki.python.org/moin/Templating) for a comprehensive list.

Some google searching also turned up a few other modules you might be
interested in:

  * [texttemplate](http://py-templates.sourceforge.net/texttemplate/index.html) (part of py-templates project)
  * [template module](http://www.embl-heidelberg.de/~chenna/pythonpages/template.html)

If you're really looking just into writing this yourself for whatever reason,
you can also dig into this Python cookbook solution [Yet Another Python
Templating Utility (YAPTU) ](http://code.activestate.com/recipes/52305/):

> _"Templating" (copying an input file to output, on the fly inserting Python
> expressions and statements) is a frequent need, and YAPTU is a small but
> complete Python module for that; expressions and statements are identified
> by arbitrary user-chosen regular-expressions._

**EDIT** : Just for the heck of it, I whipped up a severely simplistic code
sample for this. I'm sure it has bugs but it illustrates a simplified version
of the concept at least:

    
    
    #!/usr/bin/env python
    
    import sys
    import re
    
    FILE = sys.argv[1]
    
    handle = open(FILE)
    fcontent = handle.read()
    handle.close()
    
    for myexpr in re.finditer(r'\${([^}]+)}', fcontent, re.M|re.S):
        text = myexpr.group(1)
        try:
            exec text
        except SyntaxError:
            print "ERROR: unable to compile expression '%s'" % (text)
    

Tested against the following text:

    
    
    This is some random text, with embedded python like 
    ${print "foo"} and some bogus python like
    
    ${any:thing}.
    
    And a multiline statement, just for kicks: 
    
    ${
    def multiline_stmt(foo):
      print foo
    
    multiline_stmt("ahem")
    }
    
    More text here.
    

Output:

    
    
    [user@host]$ ./exec_embedded_python.py test.txt
    foo
    ERROR: unable to compile expression 'any:thing'
    ahem
    

