title: ANTLR: handle emoji characters
body: <p>I'm using following EmojiLexer.g4 together with ANTLR 4.7.2</p>
  <pre><code>lexer grammar EmojiLexer;
  
  String
      : '&quot;'   StringChar*  '&quot;'
      ;
  
  fragment StringChar
      : ~[&quot;\r\n]
      ;
  
  Punctuation
      : ( '['
        | ':'
        | ']'
        | ';'
        | '@'
        )
      ;
  
  fragment IdentifierNonDigit
      : [a-zA-Z_$\u0080-\uffff]
      ;
  
  Identifier
      : IdentifierNonDigit+
      ;
  
  Whitespace
      : [ \t]+
        -&gt; skip
      ;
  
  Newline
      : ( '\r' '\n'?
        | '\n'
        )
        -&gt; skip
      ;
  </code></pre>
  <p>and following Java code to lex a string:</p>
  <pre><code>public class EmojiTest {
  
  public static void main(String[] args) {
      final String string = &quot;[foo bar:bazz];\n&quot;
                            + &quot;\n&quot;
                            + &quot;@\&quot;emojis break it: \uD83D\uDE31\&quot;;\n&quot;
                            + &quot;\n&quot;
                            + &quot;[foo bar:bazz];\n&quot;;
      final CharStream charStream = CharStreams.fromString(string);
      final EmojiLexer lexer = new EmojiLexer(charStream);
      while (true) {
          final Token token = lexer.nextToken();
          final int type = token.getType();
          if (type &lt; 0) {
              break;
          }
  
          final int startIndex = token.getStartIndex();
          final int stopIndex = token.getStopIndex() + 1;
          System.out.println(startIndex + &quot;-&quot; + stopIndex + &quot;: &quot; + type + &quot;: &quot; + escape(string.substring(startIndex, stopIndex)));
      }
  }
  
  private static String escape(String s) {
      final StringBuilder buffer = new StringBuilder();
      for (int i = 0; i &lt; s.length(); i++) {
          final char chr = s.charAt(i);
          if (chr &gt;= 0x20 &amp;&amp; chr &lt; 0x7f) {
              buffer.append(chr);
          }
          else {
              buffer.append(&quot;\\u&quot;);
              final String hex = Integer.toHexString(chr);
              for (int j = hex.length(); j &lt; 4; j++) {
                  buffer.append('0');
              }
              buffer.append(hex);
          }
      }
      return buffer.toString();
  }
  </code></pre>
  <p>}</p>
  <p>but somehow the emoji double-character  confuses ANTLR, because the result is:</p>
  <pre><code>0-1: 2: [
  1-4: 3: foo
  5-8: 3: bar
  8-9: 2: :
  9-13: 3: bazz
  13-14: 2: ]
  14-15: 2: ;
  17-18: 2: @
  18-38: 1: &quot;emojis break it: \ud83d\ude31
  38-39: 2: &quot;
  41-42: 2: \u000a
  42-45: 3: [fo
  46-49: 3:  ba
  49-50: 2: r
  50-54: 3: :baz
  54-55: 2: z
  55-56: 2: ]
  </code></pre>
  <p>Is this a bug in my code or in ANTLR?</p>