<p>Can people point me to resources on lexing, parsing and tokenising with Python?</p>

<p>I'm doing a little hacking on an open source project (<a href="http://www.hotwire-shell.org/" rel="noreferrer">hotwire</a>) and wanted to do a few changes to the code that <a href="http://code.google.com/p/hotwire-shell/source/browse/trunk/hotwire/externals/shlex.py" rel="noreferrer">lexes</a>, <a href="http://code.google.com/p/hotwire-shell/source/browse/trunk/hotwire/command.py" rel="noreferrer">parses and tokenises</a> the commands entered into it.  As it is real working code it is fairly complex and a bit hard to work out.</p>

<p>I haven't worked on code to lex/parse/tokenise before, so I was thinking one approach would be to work through a tutorial or two on this aspect.  I would hope to learn enough to navigate around the code I actually want to alter.  Is there anything suitable out there?  (Ideally it could be done in an afternoon without having to buy and read the dragon book first ...)</p>

<p><strong>Edit:</strong> (7 Oct 2008) None of the below answers quite give what I want.  With them I could generate parsers from scratch, but I want to learn how to write my own basic parser from scratch, not using lex and yacc or similar tools.  Having done that I can then understand the existing code better.  </p>

<p>So could someone point me to a tutorial where I can build a basic parser from scratch, using just python?</p>

[{'answer_id': 36978, 'author': 'PW.', 'author_id': 927, 'author_profile': 'https://Stackoverflow.com/users/927', 'pm_score': 3, 'selected': False, 'text': '<p>Have a look at the standard module <a href="http://docs.python.org/lib/module-shlex.html" rel="nofollow noreferrer">shlex</a> and modify one copy of it to match the syntax you use for your shell, it is a good starting point</p>\n\n<p>If you want all the power of a complete solution for lexing/parsing, <a href="https://github.com/antlr/antlr4/blob/master/doc/python-target.md" rel="nofollow noreferrer">ANTLR</a> can generate python too.</p>\n'}, {'answer_id': 37245, 'author': 'nimish', 'author_id': 3926, 'author_profile': 'https://Stackoverflow.com/users/3926', 'pm_score': 2, 'selected': False, 'text': '<p>I suggest <a href="http://www.canonware.com/Parsing/" rel="nofollow noreferrer">http://www.canonware.com/Parsing/</a>, since it is pure python and you don\'t need to learn a grammar, but it isn\'t widely used, and has comparatively little documentation. The heavyweight is ANTLR and PyParsing. ANTLR can generate java and C++ parsers too, and AST walkers but you will have to learn what amounts to a new language.</p>\n'}, {'answer_id': 107187, 'author': 'Eli Bendersky', 'author_id': 8206, 'author_profile': 'https://Stackoverflow.com/users/8206', 'pm_score': 5, 'selected': False, 'text': '<p>I\'m a happy user of <a href="http://www.dabeaz.com/ply/" rel="noreferrer">PLY</a>. It is a pure-Python implementation of Lex &amp; Yacc, with lots of small niceties that make it quite Pythonic and easy to use. Since Lex &amp; Yacc are the most popular lexing &amp; parsing tools and are used for the most projects, PLY has the advantage of standing on giants\' shoulders. A lot of knowledge exists online on Lex &amp; Yacc, and you can freely apply it to PLY.</p>\n\n<p>PLY also has a good <a href="http://www.dabeaz.com/ply/ply.html" rel="noreferrer">documentation page</a> with some simple examples to get you started. </p>\n\n<p>For a listing of lots of Python parsing tools, see <a href="http://nedbatchelder.com/text/python-parsers.html" rel="noreferrer">this</a>.</p>\n'}, {'answer_id': 107207, 'author': 'nilamo', 'author_id': 19097, 'author_profile': 'https://Stackoverflow.com/users/19097', 'pm_score': 3, 'selected': False, 'text': '<p><a href="http://pygments.org/" rel="noreferrer">pygments</a> is a source code syntax highlighter written in python.  It has lexers and formatters, and may be interesting to peek at the source.</p>\n'}, {'answer_id': 137207, 'author': 'Torsten Marek', 'author_id': 9567, 'author_profile': 'https://Stackoverflow.com/users/9567', 'pm_score': 4, 'selected': False, 'text': '<p>For medium-complex grammars, <a href="https://github.com/pyparsing/pyparsing/" rel="nofollow noreferrer">PyParsing</a> is brilliant. You can define grammars directly within Python code, no need for code generation:</p>\n\n<pre><code>&gt;&gt;&gt; from pyparsing import Word, alphas\n&gt;&gt;&gt; greet = Word( alphas ) + "," + Word( alphas ) + "!" # &lt;-- grammar defined here\n&gt;&gt;&gt; hello = "Hello, World!"\n&gt;&gt;&gt;&gt; print hello, "-&gt;", greet.parseString( hello )\nHello, World! -&gt; [\'Hello\', \',\', \'World\', \'!\']\n</code></pre>\n\n<p>(Example taken from the PyParsing home page).</p>\n\n<p>With parse actions (functions that are invoked when a certain grammar rule is triggered), you can convert parses directly into abstract syntax trees, or any other representation.</p>\n\n<p>There are many helper functions that encapsulate recurring patterns, like operator hierarchies, quoted strings, nesting or C-style comments.</p>\n'}, {'answer_id': 279733, 'author': 'Tony Arkles', 'author_id': 13868, 'author_profile': 'https://Stackoverflow.com/users/13868', 'pm_score': 3, 'selected': False, 'text': '<p>Here\'s a few things to get you started (roughly from simplest-to-most-complex, least-to-most-powerful):</p>\n\n<p><a href="http://en.wikipedia.org/wiki/Recursive_descent_parser" rel="noreferrer"><a href="http://en.wikipedia.org/wiki/Recursive_descent_parser" rel="noreferrer">http://en.wikipedia.org/wiki/Recursive_descent_parser</a></a></p>\n\n<p><a href="http://en.wikipedia.org/wiki/Top-down_parsing" rel="noreferrer"><a href="http://en.wikipedia.org/wiki/Top-down_parsing" rel="noreferrer">http://en.wikipedia.org/wiki/Top-down_parsing</a></a></p>\n\n<p><a href="http://en.wikipedia.org/wiki/LL_parser" rel="noreferrer"><a href="http://en.wikipedia.org/wiki/LL_parser" rel="noreferrer">http://en.wikipedia.org/wiki/LL_parser</a></a></p>\n\n<p><a href="http://effbot.org/zone/simple-top-down-parsing.htm" rel="noreferrer"><a href="http://effbot.org/zone/simple-top-down-parsing.htm" rel="noreferrer">http://effbot.org/zone/simple-top-down-parsing.htm</a></a></p>\n\n<p><a href="http://en.wikipedia.org/wiki/Bottom-up_parsing" rel="noreferrer"><a href="http://en.wikipedia.org/wiki/Bottom-up_parsing" rel="noreferrer">http://en.wikipedia.org/wiki/Bottom-up_parsing</a></a></p>\n\n<p><a href="http://en.wikipedia.org/wiki/LR_parser" rel="noreferrer"><a href="http://en.wikipedia.org/wiki/LR_parser" rel="noreferrer">http://en.wikipedia.org/wiki/LR_parser</a></a></p>\n\n<p><a href="http://en.wikipedia.org/wiki/GLR_parser" rel="noreferrer"><a href="http://en.wikipedia.org/wiki/GLR_parser" rel="noreferrer">http://en.wikipedia.org/wiki/GLR_parser</a></a></p>\n\n<p>When I learned this stuff, it was in a semester-long 400-level university course.  We did a number of assignments where we did parsing by hand; if you want to really understand what\'s going on under the hood, I\'d recommend the same approach.</p>\n\n<p>This isn\'t the book I used, but it\'s pretty good: <a href="https://rads.stackoverflow.com/amzn/click/com/0201000229" rel="noreferrer" rel="nofollow noreferrer">Principles of Compiler Design</a>.</p>\n\n<p>Hopefully that\'s enough to get you started :)</p>\n'}, {'answer_id': 14315179, 'author': 'Saad', 'author_id': 1087841, 'author_profile': 'https://Stackoverflow.com/users/1087841', 'pm_score': 5, 'selected': False, 'text': '<p>This question is pretty old, but maybe my answer would help someone who wants to learn the basics. I find this resource to be very good. It is a simple interpreter written in python without the use of any external libraries. So this will help anyone who would like to understand the internal working of parsing, lexing, and tokenising:</p>\n\n<p>"A Simple Intepreter from Scratch in Python:" <a href="http://www.jayconrod.com/posts/37/a-simple-interpreter-from-scratch-in-python-part-1">Part 1</a>, <a href="http://www.jayconrod.com/posts/38/a-simple-interpreter-from-scratch-in-python-part-2">Part 2</a>,\n<a href="http://www.jayconrod.com/posts/39/a-simple-interpreter-from-scratch-in-python-part-3">Part 3</a>, and <a href="http://www.jayconrod.com/posts/40/a-simple-interpreter-from-scratch-in-python-part-4">Part 4</a>.</p>\n'}, {'answer_id': 54056320, 'author': 'John Greene', 'author_id': 4379130, 'author_profile': 'https://Stackoverflow.com/users/4379130', 'pm_score': 2, 'selected': False, 'text': '<p>Frederico Tomassetti had a good (but short) concise write-up to all things related from BNF to binary deciphering on:</p>\n\n<ul>\n<li>lexical, </li>\n<li>parser, </li>\n<li>abstract-syntax tree (AST), and </li>\n<li>Construct/code-generator.</li>\n</ul>\n\n<p>He even mentioned the new Parsing Expression Grammar (PEG).</p>\n\n<p><a href="https://tomassetti.me/parsing-in-python/" rel="nofollow noreferrer">https://tomassetti.me/parsing-in-python/</a></p>\n'}]
['https://Stackoverflow.com/questions/36953', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/3189/']