title: Understanding trees in ANTLR
body: <p>I'm trying to use Antlr for some text IDE-like functions -- specifically parsing a file to identify the points for code folding, and for applying syntax highlighting.</p>
  
  <p>First question - is Antlr suitable for this requirement, or is it overkill?  This <em>could</em> be achieved using regex and/or a hand-rolled parser ... but it seems that Antlr is out there to do this work for me.</p>
  
  <p>I've had a look through the ... and the excellent tutorial resource <a href="http://jumbleagilemanuals.blogspot.com/2008/03/10-steps-to-beginning-to-parsing-with.html" rel="noreferrer">here</a>.  </p>
  
  <p>I've managed to get a Java grammar built (using the <a href="http://www.antlr.org/grammar/list" rel="noreferrer">standard grammar</a>), and get everything parsed neatly into a tree.  However, I'd have expected to see elements nested within the tree.  In actual fact, everything is a child of the very top element.</p>
  
  <p>Eg.  Given:</p>
  
  <pre><code>package com.example
  public class Foo {
     String myString = "Hello World"
     // etc
  }
  </code></pre>
  
  <p>I'd have expected the tree node for Foo to be a child of the node for the package declaration.  Likewise, myString would be a child of Foo.</p>
  
  <p>Instead, I'm finding that <code>Foo</code> and <code>myString</code> (and everything else for that matter) are all children of <code>package</code>.</p>
  
  <p>Here's the relevant excerpt doing the parsing:</p>
  
  <pre><code>public void init() throws Exception {
      CharStream c = new ANTLRFileStream(
              "src/com/inversion/parser/antlr/Test.code");
  
      Lexer lexer = new JavaLexer(c);
      CommonTokenStream tokens = new CommonTokenStream(lexer);
      JavaParser parser = new JavaParser(tokens);
      parser.setTreeAdaptor(adaptor);
  
      compilationUnit_return result = parser.compilationUnit();
  }
  
  static final TreeAdaptor adaptor = new CommonTreeAdaptor() {
      public Object create(Token payload) {
          if (payload != null)
          {
              System.out.println("Create " + JavaParser.tokenNames[payload.getType()] + ":  L" + payload.getLine() + ":C" + payload.getCharPositionInLine() + " " + payload.getText());
          }
          return new CommonTree(payload);
      }
  };
  </code></pre>
  
  <p>Examining <code>result.getTree()</code> returns a <code>CommonTree</code> instance, whose children are the result of the parsing.</p>
  
  <p><strong>Expected value (perhaps incorrectly)</strong></p>
  
  <pre><code>package com.example (4 tokens)
     |
     +-- public class Foo (3 tokens)
          |
          +--- String myString = "Hello World" (4 tokens)
          +--- Comment "// etc"
  </code></pre>
  
  <p>(or something similar)</p>
  
  <p><strong>Actual value</strong> (All values are children of the root node of <code>result.getTree()</code> )</p>
  
  <pre><code>package
  com
  .
  example
  public
  class
  Foo
  String
  myString
  =
  "Hello World"
  </code></pre>
  
  <p>Is my understanding of how this should be working correct?</p>
  
  <p>I'm a complete noob at Antlr so far, and I'm finding the learning curve quite steep.</p>