title: Lexer rule to handle escape of quote with quote or backslash in ANTLR4?
body: <p>I'm trying to expand the answer to <a href="https://stackoverflow.com/questions/29800106/how-do-i-escape-an-escape-character-with-antlr-4">How do I escape an escape character with ANTLR 4?</a> to work when the &quot; can be escaped both with &quot; and \. I.e. both</p>
  <pre><code>&quot;Rob &quot;&quot;Commander Taco&quot;&quot; Malda is smart.&quot;
  </code></pre>
  <p>and</p>
  <pre><code>&quot;Rob \&quot;Commander Taco\&quot; Malda is smart.&quot;
  </code></pre>
  <p>are both valid and equivalent. I've tried</p>
  <pre><code>StringLiteral : '&quot;' ('&quot;&quot;'|'\\&quot;'|~[&quot;])* '&quot;';
  </code></pre>
  <p>but if fails to match</p>
  <pre><code>&quot;Entry Flag for Offset check and for \&quot;don't start Chiller Water Pump Request\&quot;&quot;
  </code></pre>
  <p>with the tokenizer consuming more characters than intended, i.e. consumes beyond \&quot;&quot;</p>
  <p>Anyone who knows how to define the lexer rule?</p>
  <hr />
  <p>A bit more detail...</p>
  <ul>
  <li><code>&quot;&quot;</code> succeeds</li>
  <li><code>&quot;&quot;&quot;&quot;</code> succeeds</li>
  <li><code>\&quot; &quot;</code> succeeds</li>
  <li><code>&quot;\&quot;&quot;</code> succeeds (at EOF)</li>
  <li><code>&quot;\&quot;&quot;\n&quot;&quot;</code> fails (it greedily pulls in the <code>\n</code> and <code>&quot;</code></li>
  </ul>
  <p>Example: (text.txt)</p>
  <pre><code>&quot;&quot;
  &quot;&quot;&quot;&quot;
  &quot;\&quot; &quot;
  &quot;\&quot;&quot;
  &quot;&quot;
  </code></pre>
  <pre><code>grun test tokens -tokens &lt; test.txt
  </code></pre>
  <pre><code>line 5:1 token recognition error at: '&quot;'
  [@0,0:1='&quot;&quot;',&lt;StringLiteral&gt;,1:0]
  [@1,2:2='\n',&lt;'
  '&gt;,1:2]
  [@2,3:6='&quot;&quot;&quot;&quot;',&lt;StringLiteral&gt;,2:0]
  [@3,7:7='\n',&lt;'
  '&gt;,2:4]
  [@4,8:12='&quot;\&quot; &quot;',&lt;StringLiteral&gt;,3:0]
  [@5,13:13='\n',&lt;'
  '&gt;,3:5]
  [@6,14:19='&quot;\&quot;&quot;\n&quot;',&lt;StringLiteral&gt;,4:0]
  [@7,21:20='&lt;EOF&gt;',&lt;EOF&gt;,5:2]
  </code></pre>
  <p><code>\&quot;&quot;</code> and <code>&quot;&quot;&quot;</code> at the end of a StringListeral are not being handled the same.</p>
  <p>Here's the ATN for that rule:</p>
  <p><a href="https://i.stack.imgur.com/Ozi27.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Ozi27.png" alt="enter image description here" /></a></p>
  <p>From this diagram it's not clear why they should be handled differently.  They appear to be parallel constructs.</p>
  <hr />
  <h2>More research</h2>
  <p>Test Grammar (small change to simplify ATN):</p>
  <pre><code>grammar test
      ;
  
  start: StringLiteral (WS? StringLiteral)+;
  
  StringLiteral: '&quot;' ( (('\\' | '&quot;') '&quot;') | ~[&quot;])* '&quot;';
  WS:            [ \t\n\r]+;
  </code></pre>
  <p>The ATN for StringLiteral in this grammar:</p>
  <p><a href="https://i.stack.imgur.com/mL9lX.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/mL9lX.png" alt="ATN for Stringliteral" /></a></p>
  <p>OK, let's walk through this ATN with the input <code>&quot;\&quot;&quot;\n&quot;</code></p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th style="text-align: left;">unconsumed input</th>
  <th>transition</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td style="text-align: left;">&quot;\&quot;&quot;\n&quot;</td>
  <td>1 -ε-&gt; 5</td>
  </tr>
  <tr>
  <td style="text-align: left;">&quot;\&quot;&quot;\n&quot;</td>
  <td>5 -&quot;-&gt; 11</td>
  </tr>
  <tr>
  <td style="text-align: left;">\&quot;&quot;\n&quot;</td>
  <td>11 -ε-&gt; 9</td>
  </tr>
  <tr>
  <td style="text-align: left;">\&quot;&quot;\n&quot;</td>
  <td>9 -ε-&gt; 6</td>
  </tr>
  <tr>
  <td style="text-align: left;">\&quot;&quot;\n&quot;</td>
  <td>6 -\-&gt; 7</td>
  </tr>
  <tr>
  <td style="text-align: left;">&quot;&quot;\n&quot;</td>
  <td>7 -&quot;-&gt; 10</td>
  </tr>
  <tr>
  <td style="text-align: left;">&quot;\n&quot;</td>
  <td>10 -ε-&gt; 13</td>
  </tr>
  <tr>
  <td style="text-align: left;">&quot;\n&quot;</td>
  <td>13 -ε-&gt; 11</td>
  </tr>
  <tr>
  <td style="text-align: left;">&quot;\n&quot;</td>
  <td>11 -ε-&gt; 12</td>
  </tr>
  <tr>
  <td style="text-align: left;">&quot;\n&quot;</td>
  <td>12 -ε-&gt; 14</td>
  </tr>
  <tr>
  <td style="text-align: left;">&quot;\n&quot;</td>
  <td>14 -&quot;-&gt; 15</td>
  </tr>
  <tr>
  <td style="text-align: left;">\n&quot;</td>
  <td>15 -ε-&gt; 2</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>We should reach State <code>2</code> with the <code>&quot;</code> before the <code>\n</code>, which would be the desired behavior.</p>
  <p>Instead, we see it continue on to consume the <code>\n</code> and the next <code>&quot;</code></p>
  <pre><code>line 2:1 token recognition error at: '&quot;'
  [@0,0:5='&quot;\&quot;&quot;\n&quot;',&lt;StringLiteral&gt;,1:0]
  [@1,7:6='&lt;EOF&gt;',&lt;EOF&gt;,2:2]
  </code></pre>
  <p>In order for this to be valid, there must be a path from state 11 to state 2 that consumes a <code>\n</code> and a <code>&quot;</code> (and I'm not seeing it)</p>
  <p>Maybe I'm missing something, but it's looking more and more like a bug to me.</p>