title: javacc bug ?? it gave me a token I never wrote in the input
body: <p>I gave this input string to my javacc-based parser: </p>
  
  <pre><code>"  SELECT  a\n.run_id,\nx.*"
  </code></pre>
  
  <p>but it gave me an exception:</p>
  
  <pre><code>net.sf.jsqlparser.parser.ParseException: Encountered " ";" "; "" at line 3, column 1.
  Was expecting one of:
      "*" ...
      "*" ...
  
      at net.sf.jsqlparser.parser.CCJSqlParser.generateParseException(CCJSqlParser.java:14079)
      at net.sf.jsqlparser.parser.CCJSqlParser.jj_consume_token(CCJSqlParser.java:13929)
      at net.sf.jsqlparser.parser.CCJSqlParser.SelectItem(CCJSqlParser.java:2092)
      at net.sf.jsqlparser.parser.CCJSqlParser.SelectItemsList(CCJSqlParser.java:2017)
      at net.sf.jsqlparser.parser.CCJSqlParser.PlainSelect(CCJSqlParser.java:1676)
      at net.sf.jsqlparser.parser.CCJSqlParser.SelectBody(CCJSqlParser.java:1574)
      at net.sf.jsqlparser.parser.CCJSqlParser.Select(CCJSqlParser.java:1550)
      at net.sf.jsqlparser.parser.CCJSqlParser.SingleStatement(CCJSqlParser.java:98)
      at net.sf.jsqlparser.parser.CCJSqlParser.Statements(CCJSqlParser.java:186)
      at com.paypal.dt.roe.parser.SqlConvertor.convert(SqlConvertor.java:286)
      at com.paypal.dt.roe.parser.SqlConvertorTest.test1(SqlConvertorTest.java:247)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
      at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
      at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
      at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:74)
      at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:83)
      at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:72)
      at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:231)
      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
      at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
      at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
      at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
      at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
      at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
      at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
      at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:71)
      at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
      at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:174)
      at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
      at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
  </code></pre>
  
  <p>this is really surprising, since I did not write the ";" char at all!!</p>
  
  <p>in fact , if I remove all the "\n" or do some other format tweaks, the error is gone. </p>
  
  <p>I enabled tokenmanager logging in javacc, and here is the relevant lines in logging:</p>
  
  <pre><code>$ grep -B 4 FOUND /tmp/a
     Starting NFA to match one of : { &lt;S_IDENTIFIER&gt; }
  Current character : 32 (32) at line 1 column 9
     Currently matched the first 6 characters as a "SELECT" token.
     Putting back 1 characters into the input stream.
  ****** FOUND A "SELECT" MATCH (SELECT) ******
  --
     Starting NFA to match one of : { &lt;S_IDENTIFIER&gt; }
  Current character : 10 (10) at line 1 column 12
     Currently matched the first 1 characters as a &lt;S_IDENTIFIER&gt; token.
     Putting back 1 characters into the input stream.
  ****** FOUND A &lt;S_IDENTIFIER&gt; MATCH (a) ******
  --
     Starting NFA to match one of : { &lt;S_DOUBLE&gt; }
  Current character : 114 (114) at line 2 column 2
     Currently matched the first 1 characters as a "." token.
     Putting back 1 characters into the input stream.
  ****** FOUND A "." MATCH (.) ******
  --
     Possible kinds of longer matches : { &lt;K_TIMEUNIT&gt;, &lt;S_IDENTIFIER&gt; }
  Current character : 44 (44) at line 2 column 8
     Currently matched the first 6 characters as a &lt;S_IDENTIFIER&gt; token.
     Putting back 1 characters into the input stream.
  ****** FOUND A &lt;S_IDENTIFIER&gt; MATCH (run_id) ******
  --
                      Return: RelObjectName(LOOKAHEAD SUCCEEDED)
  Current character : 44 (44) at line 2 column 8
     No more string literal token matches are possible.
     Currently matched the first 1 characters as a "," token.
  ****** FOUND A "," MATCH (,) ******
  --
  Skipping character : 10 (10)
  Current character : 59 (59) at line 3 column 1
     No more string literal token matches are possible.
     Currently matched the first 1 characters as a ";" token.
  ****** FOUND A ";" MATCH (;) ******
  --
     Starting NFA to match one of : { &lt;S_IDENTIFIER&gt; }
  Current character : 46 (46) at line 3 column 3
     Currently matched the first 1 characters as a &lt;S_IDENTIFIER&gt; token.
     Putting back 1 characters into the input stream.
  ****** FOUND A &lt;S_IDENTIFIER&gt; MATCH (x) ******
  </code></pre>