title: ANTLR: Unicode Character Scanning
body: <p>Problem: Can't get Unicode character to print correctly.</p>
  
  <p>Here is my grammar:</p>
  
  <pre><code>options { k=1; filter=true;
   // Allow any char but \uFFFF (16 bit -1)
  charVocabulary='\u0000'..'\uFFFE'; 
  }
  
  ANYCHAR :'$'
  |    '_' { System.out.println("Found underscore: "+getText()); }
  |    'a'..'z' { System.out.println("Found alpha: "+getText()); }
  |    '\u0080'..'\ufffe' { System.out.println("Found unicode: "+getText()); }
  ; 
  </code></pre>
  
  <p>Code snippet of main method invoking the lexer:</p>
  
  <pre><code>public static void main(String[] args) {
  SimpleLexer simpleLexer = new SimpleLexer(System.in);
  while(true) {
  try {
  Token t = simpleLexer.nextToken();
  System.out.println("Token : "+t);
  
  } catch(Exception e) {}
  
  }
  }
  </code></pre>
  
  <p>For input <strong>"ठ"</strong>, I'm getting the following output :</p>
  
  <pre><code>Found unicode: 
  Token : ["à",&lt;5&gt;,line=1,col=7]
  Found unicode: 
  Token : ["¤",&lt;5&gt;,line=1,col=8]
  Found unicode:  
  Token : [" ",&lt;5&gt;,line=1,col=9]
  </code></pre>
  
  <p>It appears that the lexer is treating Unicode char "ठ" as three separate character. My aim is to scan and print "ठ".</p>