title: Java String Tokenizing for a Programming Language Parser
body: <p>A few months ago, I thought it would be cool to make my own programming language. So I did. I named it <a href="https://github.com/nrubin29/Pogo" rel="nofollow">Pogo</a>. If you look at the code, you'll see a ton of <code>String#split</code>. Here's some pseudocode that describes how I parse a method header:</p>
  
  <p>Given <code>method public void main()</code>, I look to see if <code>String#startsWith("method")</code>. If so, I <code>String#split(" ")</code> and parse the second word (public) to a visibility. I treat the third word (void) as the return type, and check to see if it's a primitive or class. Finally, I treat the last word as the name and parse any method parameters properly.</p>
  
  <p>This method works, but I know it's wrong. You're supposed to tokenize everything and handle it like that. However, I don't quite understand how that would work. I know I'd use Java's <code>StringTokenizer</code>, but how would I parse a method header? Given <code>method public void main()</code>, the first token would be method. I'd know I'm working with a method, but I can't instantiate my <code>Method</code> class because I don't have all the information yet. It seems like I'd have to declare a loop for the <code>StringTokenizer</code> and have tons of variables outside of it, which almost seems worse than what I have now.</p>
  
  <p><strong>tl;dr</strong>: How should I use String tokenization to parse a custom programming language. I'm not really looking for code, more like pseudocode or ideas.</p>
  
  <p>Thanks!</p>