title: Lexical analyzer (java) for HTML Markdown source code
body: <p>I do not even know where to begin writing the character-by-character lexical analyzer. I wrote BNF grammar rules for a Markdown language (specifically, HTML) based on rules and specifics I was given, so none should need to be added. I now have to design and implement a character-by-character lexical analyzer that partitions the lexemes of a source file in my Markdown language into tokens. Here is my BNF GRAMMAR:</p>
  
  <p>Terminals:</p>
  
  <pre><code>#DOCUMENT BEGIN,
  #DOCUMENT END
  #HEAD BEGIN,
  #HEAD END,
  #TITLE BEGIN,
  #TITLE END,
  #PARAGRAPH BEGIN,
  #PARAGRAPH END,
  #BOLD BEGIN,
  #BOLD END,
  #ITALICS BEGIN,
  #ITALICS END,
  #LIST BEGIN,
  #LIST END,
  #ITEM BEGIN,
  #ITEM END,
  #LINK BEGIN,
  #TEXT,
  #ADDRESS,
  #LINK END,
  #DEFINE BEGIN,
  #NAME,
  #VALUE,
  #DEFINE END,
  #USE BEGIN,
  #USE END
  </code></pre>
  
  <p><em><strong>Note that these terminals are not case sensitive.</em></strong></p>
  
  <p>Non-Terminals:</p>
  
  <pre><code>&lt;document&gt; ::= #DOCUMENT BEGIN &lt;macro-­‐define&gt; &lt;head&gt; &lt;body&gt; #DOCUMENT END
  
  &lt;head&gt; ::= #HEAD BEGIN &lt;title&gt; #HEAD END | ε
  
  &lt;title&gt; ::= #TITLE BEGIN &lt;text&gt; #TITLE END | ε
  
  &lt;body&gt; ::= &lt;inner-­‐text&gt; &lt;body&gt;
             | &lt;paragraph&gt; &lt;body&gt;
             | &lt;bold&gt; &lt;body&gt;
             | &lt;italics&gt; &lt;body&gt;
             | &lt;list&gt; &lt;body&gt;
             | ε
  
  &lt;paragraph&gt; ::= #PARAGRAPH BEGIN &lt;macro-­‐define&gt; &lt;inner-­‐paragraph&gt; #PARAGRAPH END
  
  &lt;inner-­‐paragraph&gt; ::= &lt;inner-­‐text&gt; &lt;inner-­‐paragraph&gt;
                        | &lt;bold&gt; &lt;inner-­‐paragraph&gt;
                        | &lt;italics&gt; &lt;inner-­‐paragraph&gt;
                        | &lt;list&gt; &lt;inner-­‐paragraph&gt;
                        | ε
  
  &lt;inner-­‐text&gt; ::= &lt;macro-­‐use&gt; &lt;inner-­‐text&gt;
                    | &lt;text&gt; &lt;inner-­‐text&gt;
                    | ε
  
  &lt;macro-­‐define&gt; ::= #DEFINE BEGIN #NAME &lt;text&gt; #VALUE &lt;body&gt; #DEFINE END &lt;macro-­‐define&gt;
                      | ε
  
  &lt;macro-­‐use&gt; ::= #USE BEGIN &lt;text&gt; #USE END | ε
  
  &lt;bold&gt; ::= #BOLD BEGIN &lt;macro-­‐define&gt; &lt;inner-­‐text&gt; #BOLD END
  
  &lt;italics&gt; ::= #ITALICS BEGIN &lt;macro-­‐define&gt; &lt;inner-­‐text&gt; #ITALICS END
  
  &lt;link&gt; ::= #LINK BEGIN #TEXT &lt;text&gt; #ADDRESS &lt;text&gt; #LINK END
  
  &lt;list&gt; ::= #LIST BEGIN #ITEM BEGIN &lt;macro-­‐define&gt; &lt;inner-­‐list&gt; #ITEM END &lt;list-­‐items&gt; #LIST END
  
  &lt;list-­‐items&gt; ::= #ITEM BEGIN &lt;macro-­‐define&gt; &lt;inner-­‐list&gt; #ITEM END &lt;list-­‐items&gt; | ε
  
  &lt;inner-­‐list&gt; ::= | &lt;bold&gt; &lt;inner-­‐list&gt;
                    | &lt;italics&gt; &lt;inner-­‐list&gt;
                    | &lt;list&gt;&lt;inner-­‐list&gt;
                    | &lt;inner-­‐text&gt; &lt;inner-­‐list&gt;
                    | ε
  
  &lt;text&gt; ::= Any plain text | ε
  </code></pre>
  
  <p>We can assume that HTML characters such as "&lt;", ">", "&amp;", and "/" do not appear in any of the text in the source file. We can also assume that "#" only appears prior to one of our Markdown annotations (e.g., #DOCUMENT). I think it would be best to have separate Java classes to represent token objects such as: <em>DocumentBegin, DocumentEnd, ParagraphBegin, ParagraphEnd, etc.</em> Any lexical errors encountered (e.g., #DOC BEGIN) should be reported as output to the console with as much error info as possible. The compiler should exit after first error is encountered. If an error is encountered, no output file should be created.</p>
  
  <p>My problem is, I know what a lexical analyzer is supposed to do, but honestly I have no clue where to begin with coding/implementing. If you need more explanation of what the problem is asking, please just ask and I can do my best to explain. This was one part of a big project we had due for my class. I was unable to complete this part and lost a lot of points, but now I just need to understand it so once we are tested on it, I won't be as lost.</p>