   Generic Tokenizer
  I''m interested in writing a class/set of classes that would be able to take a file and tokenize it according to a non-hard-coded grammar.  That is, something which could load in and parse my settings files as well as my game scripts, particle system scripts, etc.  Obviously, these would have similar, yet different grammar rules, and I''d like to encapsulate the common code.To this end, I think I''d need to come up with some sort of CGrammar class, which would hold the rules for what each token can be composed of, what can follow what, etc., for each file type, and then pass that to the CTokenizer class, which would then be able to apply it to the given file; the CTokenizer would build up a tree of tokens, which could then be queried by the calling class.Thoughts on this design?  How would you encapsulate the grammars?  (I''m thinking a list of tokenrules, each of which has a callback verification function and a list of allowable following tokens, but that seems like it could get ugly quickly.)  Has this sort of thing been done before?  Where I can see the code? Thanks,-Odd the Hermit(The idea for this came mostly from the Particle System tokenizer in Mason McCuskey''s "Special Effects Game Programming with Direct X", chapter 19, published by Premier Press.) 
  Are you familar with boost?  They added the SpiritX library recently, which is an inf(LL) parser they you code-up in C++ using xBNF-like syntax.   - Magmai Kai Holmlor"No, his mind is not for rent to any god nor government" -  Rush, Tom Sawyer[Look for information | GDNet Start Here | GDNet Search Tool | GDNet FAQ | MSDN RTF[L] | SGI STL Docs | STFW | Asking Smart Questions ][Free C++ Libraries | Boost | ACE | Loki | MTL | Blitz++ | wxWindows| Spirit(xBNF)][Free C Libraries | zlib ];  The Design Patterns book discusses an Interpreter pattern that might be of some use here. Oftentimes with patterns in that book you have to work with them awhile to extend them so they work for your needs better, but it might be a starting point. 