<p>I've been given a job of 'translating' one language into another. The source is too flexible (complex) for a simple line by line approach with regex. Where can I go to learn more about lexical analysis and parsers?</p>

[{'answer_id': 34085, 'author': 'Greg Hewgill', 'author_id': 893, 'author_profile': 'https://Stackoverflow.com/users/893', 'pm_score': 2, 'selected': False, 'text': '<p>I\'ve recently been working with <a href="http://www.dabeaz.com/ply/" rel="nofollow noreferrer">PLY</a> which is an implementation of lex and yacc in Python. It\'s quite easy to get started with it and there are some simple examples in the documentation.</p>\n\n<p>Parsing can quickly become a very technical topic and you\'ll find that you probably won\'t need to know all the details of the parsing algorithm if you\'re using a parser builder like PLY.</p>\n'}, {'answer_id': 34086, 'author': 'jfs', 'author_id': 718, 'author_profile': 'https://Stackoverflow.com/users/718', 'pm_score': 3, 'selected': False, 'text': '<p>Try <a href="http://www.antlr.org/" rel="nofollow noreferrer">ANLTR</a>:</p>\n\n<blockquote>\n  <p>ANTLR, ANother Tool for Language\n  Recognition, is a language tool that\n  provides a framework for constructing\n  recognizers, interpreters, compilers,\n  and translators from grammatical\n  descriptions containing actions in a\n  variety of target languages.</p>\n</blockquote>\n\n<p>There\'s a <a href="http://www.pragprog.com/titles/tpantlr/the-definitive-antlr-reference" rel="nofollow noreferrer">book</a> for it also.</p>\n\n<p><img src="https://i.stack.imgur.com/i8AUE.jpg" alt="alt text"></p>\n'}, {'answer_id': 34089, 'author': 'tsellon', 'author_id': 3575, 'author_profile': 'https://Stackoverflow.com/users/3575', 'pm_score': 1, 'selected': False, 'text': '<p>If you prefer Java based tools, the Java Compiler Compiler, JavaCC, is a nice parser/scanner.  It\'s config file driven, and will generate java code that you can include in your program.  I haven\'t used it a couple years though, so I\'m not sure how the current version is.  You can find out more here: <a href="https://javacc.dev.java.net/" rel="nofollow noreferrer">https://javacc.dev.java.net/</a></p>\n'}, {'answer_id': 34101, 'author': 'Matt Cummings', 'author_id': 828, 'author_profile': 'https://Stackoverflow.com/users/828', 'pm_score': 4, 'selected': False, 'text': '<p>If you want to get "emotional" about the subject, pick up a copy of "<a href="http://en.wikipedia.org/wiki/Compilers:_Principles,_Techniques,_and_Tools" rel="noreferrer">The Dragon Book</a>."  It is <em>usually</em> the text in a compiler design course.  It will definitely meet your need "learn more about lexical analysis and parsers" as well as a bunch of other <strong>fun</strong> stuff!  </p>\n\n<p>IMH(umble)O, save yourself an arm and/or leg and buy an older edition - it will fill your information desires.</p>\n'}, {'answer_id': 34104, 'author': 'hoyhoy', 'author_id': 3499, 'author_profile': 'https://Stackoverflow.com/users/3499', 'pm_score': 1, 'selected': False, 'text': "<p>flex and bison are the new lex and yacc though.  The syntax for BNF is often derided for being a bit obtuse.  Some have moved to ANTLR and Ragel for this reason.</p>\n\n<p>If you're not doing much translation, you may one to pull a one-off using multiline regexes with Perl or Ruby.  Writing a compatible BNF grammar for an existing language is not a task to be taken lightly.  </p>\n\n<p>On the other hand, it is entirely possible to leverage any given language's .l and .y files if they are available as open source.  Then, you could construct new code from an existing parse tree.</p>\n"}, {'answer_id': 34118, 'author': 'svrist', 'author_id': 86, 'author_profile': 'https://Stackoverflow.com/users/86', 'pm_score': 1, 'selected': False, 'text': "<p>Lexing/Parsing + typecheck + code generation is a great CS exercise I would recommend it to anyone wanting a solid basis, so I'm all for the Dragon Book</p>\n"}, {'answer_id': 34148, 'author': 'tamberg', 'author_id': 3588, 'author_profile': 'https://Stackoverflow.com/users/3588', 'pm_score': 3, 'selected': False, 'text': '<p>Niklaus Wirth\'s book "Compiler Construction" (available as a free PDF)\n<a href="http://www.google.com/search?q=wirth+compiler+construction" rel="noreferrer">http://www.google.com/search?q=wirth+compiler+construction</a></p>\n'}, {'answer_id': 97281, 'author': 'user11318', 'author_id': 11318, 'author_profile': 'https://Stackoverflow.com/users/11318', 'pm_score': 1, 'selected': False, 'text': '<p>Yet another textbook to consider is <a href="https://rads.stackoverflow.com/amzn/click/com/0126339511" rel="nofollow noreferrer" rel="nofollow noreferrer">Programming Language Pragmatics</a>.  I prefer it over the Dragon book, but YMMV.</p>\n\n<p>If you\'re using Perl, yet another tool to consider is <a href="http://search.cpan.org/dist/Parse-RecDescent/lib/Parse/RecDescent.pm" rel="nofollow noreferrer">Parse::RecDescent</a>.</p>\n\n<p>If you just need to do this translation once and don\'t know anything about compiler technology, I would suggest that you get as far as you can with some fairly simplistic translations and then fix it up by hand.  Yes, it is a lot of work.  But it is less work than learning a complex subject and coding up the right solution for one job.  That said, you should still learn the subject, but don\'t let not knowing it be a roadblock to finishing your current project.</p>\n'}, {'answer_id': 153115, 'author': 'Matt', 'author_id': 17693, 'author_profile': 'https://Stackoverflow.com/users/17693', 'pm_score': 1, 'selected': False, 'text': '<p>I found this site helpful:</p>\n\n<p><a href="http://ds9a.nl/lex-yacc/cvs/output/lexyacc.html#toc1" rel="nofollow noreferrer">Lex and YACC primer/HOWTO</a></p>\n\n<p>The first time I used lex/yacc was for a relatively simple project. This tutorial was all I really needed. When I approached more complex projects later, the familiarity I had from this tutorial and a simple project allowed me to build something fancier.</p>\n'}, {'answer_id': 234811, 'author': 'jacobko', 'author_id': 1318, 'author_profile': 'https://Stackoverflow.com/users/1318', 'pm_score': 1, 'selected': False, 'text': '<p>After taking (quite) a few compilers classes, I\'ve used both <a href="https://rads.stackoverflow.com/amzn/click/com/0321486811" rel="nofollow noreferrer" rel="nofollow noreferrer">The Dragon Book</a> and <a href="https://rads.stackoverflow.com/amzn/click/com/155860698X" rel="nofollow noreferrer" rel="nofollow noreferrer">C&amp;T</a>. I think C&amp;T does a far better job of making compiler construction digestible. Not to take anything away from The Dragon Book, but I think C&amp;T is a far more practical book.</p>\n\n<p>Also, if you like writing in Java, I recommend using <a href="http://jflex.de/" rel="nofollow noreferrer">JFlex</a> and <a href="http://byaccj.sourceforge.net/" rel="nofollow noreferrer">BYACC/J</a> for your lexing and parsing needs.</p>\n'}, {'answer_id': 234863, 'author': 'wnoise', 'author_id': 15464, 'author_profile': 'https://Stackoverflow.com/users/15464', 'pm_score': 2, 'selected': False, 'text': '<p>Lots of people have recommended books.  For many these are much more useful in a structured environment with assignments and due dates and so forth.  Even if not, having the material presented in a different way can help greatly.</p>\n\n<p>(a) Have you considered going to a school with a decent CS curriculum?<br>\n(b) There are lots of online lectures, such as <a href="http://ocw.mit.edu/OcwWeb/web/home/home/index.htm" rel="nofollow noreferrer">MIT\'s Open Courseware</a>.  Their <a href="http://ocw.mit.edu/OcwWeb/Electrical-Engineering-and-Computer-Science/" rel="nofollow noreferrer">EE/CS section</a> has many courses that touch on parsing, though I can\'t see any on parsing per se.  It\'s typically introduced as one of the first theory courses as language classification and automata is at the heart of much of CS theory.</p>\n'}, {'answer_id': 237911, 'author': 'Gio', 'author_id': 3477, 'author_profile': 'https://Stackoverflow.com/users/3477', 'pm_score': 1, 'selected': False, 'text': '<p><a href="http://dickgrune.com/Books/PTAPG_1st_Edition/" rel="nofollow noreferrer">Parsing Techniques - A Practical Guide</a>\nBy Dick Grune and Ceriel J.H. Jacobs </p>\n\n<p>This book (freely available as PDF) gives an extensive overview of different parsing techniques/algorithms. If you really want to understand the different parsing algorithms, this IMO is a better reference than the Dragon Book (as Parsing Techniques focuses entirely on parsing, while the Dragon Book covers parsing only as one - although important - part of the compiler construction process).</p>\n'}]
['https://Stackoverflow.com/questions/34081', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/3577/']