Python parser for Python-like language
Question: I'm looking to write a Python import filter or preprocessor for source files
that are essentially Python with extra language elements. The goal is to read
the source file, parse it to an abstract syntax tree, apply some transforms in
order to implement the new parts of the language, and write valid Python
source which can then be consumed by CPython. I want to write this thing in
Python and am looking for the best parser for the task.

The parser built in to Python is not appropriate because it requires the
source files be actual Python, which these will not be. There are tons of
parsers (or parser generators) that will work with Python, but it's hard to
tell which is the best for my needs without a whole bunch of research.

In summary, my requirements are:

  1. Parser is written in Python or has Python bindings.
  2. Comes with a Python grammar that I can tweak, or can easily consume a tweakable Python grammar available elsewhere (such as <http://docs.python.org/reference/grammar.html>).
  3. Can re-serialize the AST after transforming it.
  4. Should not be too horrific to work with API-wise.

Any suggestions?


Answer: The first thing that comes to mind is
[`lib2to3`](http://docs.python.org/library/2to3.html#module-lib2to3). It is a
complete pure-Python implementation of a Python parser. It reads a Python
grammar file and parses Python source files according to this grammar. It
offers a great infrastructure for performing AST manipulations and writing
back nicely formatted Python code -- after all it's purpose is to transform
between two Python-like languages with slightly different grammars.

Unfortunately it's lacking documentation and doesn't guarantee a stable
interface. There are projects that build on top of `lib2to3` nevertheless, and
the [source code](http://hg.python.org/cpython/file/80ab0b13eb04/Lib/lib2to3)
is quite readable. If API stability is an issue, you can just fork it.

