   Text Parser framework ideas please!
  Hello,I have my own DX GUI. At present anything entered into the input box is simply added to 'chatbox'.I now need to parse the inputted text. This arrives as a std::string, in a single line. Obviously I cannot use select/case on this, so what is the best way of parsing this text data?I want to be able to enter commands like this:/tell player hello!<- standard message formathello!   <- shortcut to local chat/who<- command#wireframe    <- system command for debugging/GMsThoughts appreciatedSimon   
  I think what you are asking for are Flex and Bison. Maybe only Bison. Essentially you want to make a parser, but these kinds of tools are usually used for this kind of task. Documentation or someone else might be able to explain it better. FlexBisonOtherwise for a simple solution you have to manually tokenize the string and look for the patterns.   ;  You might wanna check out boost::spiritCheers!Eric   ;  Quake source code. It shows you how to do this command/chat console and how to parse up the text and execute the commands.Basically, you do this "class" that has the name and function to call.struct command { string name; void (*callback)(vector<string> & args);};Then you have function to register these commands by name and with the callback pointer, and these are stored to list.Now when you wanna execute some command, started with '/', you just search it from the list and call the callback.Or send it to chat it its not a command, or doesnt start with '/'.For parsing the string itself, all you need to do is a simple loop :- Skip whitespace- Check '/'- Begin Loop- - Skip whitesspace- - Read alphabets- - Loopch.   ;  Manually writing the parser is easier than using a parser generator for this very simple task.Just split the string at the leftmost space, look up the left part in a map from string to operation and call the operation with the right part of the string as parameter. If you don't find the string in the map call the default operation with the original input string as parameter.This behaves a bit different from WoW or Quake as it just uses local chat when you enter "/thiscommanddoesntexist" instead of giving an error message.   ;  In c++ I did this with boost::spirit. In C# with Regexes. The actual design is a pluggable interpreter something like this:Interface - Defines the common interface to the interpreter, Parse, Query for command's existance, stuff to get plugged commands + argument types.cmdset - Takes boost::function/Delegate objects + string to define the callback associated with a command (the first "word").cmdgroup - Holds multiple interfaces, trying them in order.cmdfallthrough - Holds a single boost::function/Delegate, and passes line as string to it no matter what.cmdbreakout - Holds a prefix string (like "/" or "#") and an Interface. Any command that starts with the string is matched, and sent to the Interface (with the prefix removed).So for your chat it's built like so:ChatInterpreter  - group    - breakout ("/") - set (slash commands)    - breakout ("#") - set (debugging commands)    - fallthrough (send to chat)Works really well for me.   ;  I am tempted to write something for this manually, since the third party stuff looks unnecessarily sophisticated.How hard can it be? <deja vu>:)   ;  It's pretty easy. Here's something I coded up in python just now which should demonstrate the idea:It's not written pythony so it's easier to port over to C/C++# token types - enum in C ... I'm using strings as the values because i'm lazyTT_NONE = "TT_NONE"TT_BEGIN_COMMAND = "TT_BEGIN_COMMAND"TT_STRING_LITERAL = "TT_STRING_LITERAL"TT_LEFT_PARENTHESIS = "TT_LEFT_PARENTHESIS"TT_RIGHT_PARENTHESIS = "TT_RIGHT_PARENTHESIS"TT_IDENTIFIER = "TT_IDENTIFIER"TT_NUMBER = "TT_NUMBER"TT_END_OF_INPUT = "TT_END_OF_INPUT"TT_UNKNOWN = "TT_UNKNOWN"CT_SAYTO = "CT_SAYTO"CT_QUIT = "CT_QUIT"CT_WAVE = "CT_WAVE"CT_GIVEGOLD = "CT_GIVEGOLD"CT_NONE = "CT_NONE"class ParserError:    def __init__(self, message):   self.message = message    def __str__(self):   return self.messageclass Token:    def __init__(self, token_type=TT_NONE, data=""):   self.type = token_type   self.data = dataclass Command:    def __init__(self, command_type):   self.type = command_typeclass Tokenizer:    def __init__(self, data):   self.position = 0   self.character = ""   self.data = data   self.token = None   self.GetNextCharacter()    def GetNextCharacter(self):   if self.position < len(self.data):  self.character = self.data[self.position]  self.position += 1  return self.character   else:  self.character = ""  return ""    def SkipWhitespace(self):   while self.character != "" and self.character.isspace():  self.GetNextCharacter()    def GetIdentifier(self):   done = False   string = ""   while self.character != "" and not done:  # allows alpha or numbers  if not self.character.isalnum(): done = True  else: string += self.character self.GetNextCharacter()   self.SkipWhitespace()   return Token(TT_IDENTIFIER, string)    def GetNumber(self):   done = False   string = ""   pointCount = 0   while self.character != "" and not done:  accept = True  if self.character == "." and pointCount > 0: done = True accept = False  elif self.character == ".": pointCount += 1  elif not self.character.isdigit(): done = True accept = False  if accept: string += self.character self.GetNextCharacter()   self.SkipWhitespace()return Token(TT_NUMBER, string)    def GetStringLiteral(self, endChar):   done = False   string = ""   # skip the opening quote   self.GetNextCharacter()   while self.character != "" and not done and self.character != endChar:  string += self.character  self.GetNextCharacter()   self.GetNextCharacter() # skip the closing quote   self.SkipWhitespace() return Token(TT_STRING_LITERAL, string) def GetNextToken(self):   if self.character == "":  self.token = Token(TT_END_OF_INPUT)   elif self.character == '/':  c = self.character  self.GetNextCharacter()  # if you don't want to allow whitespace after the / remove this line  self.SkipWhitespace()  self.token = Token(TT_BEGIN_COMMAND, c)   elif self.character == '"' or self.character == "'":  self.token = self.GetStringLiteral(self.character) elif self.character.isdigit():  self.token = self.GetNumber()   elif self.character.isalpha():  self.token = self.GetIdentifier()   else:  self.token = Token(TT_UNKNOWN)   return self.token    def GetNextAsRawStringLiteral(self):   """ just returns the remaining portion of the input as a string literal token """   # self.data[self.position:] is a slice. it returns the portion of the string after self.position inclusive   data = self.character + self.data[self.position:]   self.token = Token(TT_STRING_LITERAL, data)   self.position = len(self.data)   return self.tokenclass Command:    def __init__(self, command_type=CT_NONE):   self.type = command_type   self.data = [None, None]    def ParseWave(self, tokenizer):   self.type = CT_WAVE   tokenizer.GetNextToken()    def ParseQuit(self, tokenizer):   self.type = CT_QUIT   tokenizer.GetNextToken()    def ParseGiveGold(self, tokenizer):   token = tokenizer.GetNextToken()   if token.type != TT_IDENTIFIER:  raise ParserError("Expected name")   else:  # remember name  self.data[0] = token.data  token = tokenizer.GetNextToken()  if token.type != TT_NUMBER: raise ParserError("Expected amount!")  else: self.type = CT_GIVEGOLD self.data[1] = token.data tokenizer.GetNextToken()    def ParseSayTo(self, tokenizer):   token = tokenizer.GetNextToken()   if token.type != TT_IDENTIFIER:  raise ParserError("Expected name")   else:  # remember name  self.data[0] = token.data  token = tokenizer.GetNextAsRawStringLiteral()  if token.type != TT_STRING_LITERAL: raise ParserError("Expected string! got " + token.type + " '" + token.data + "'")  else: self.type = CT_SAYTO self.data[1] = token.data tokenizer.GetNextToken()def ParseCommand(self, tokenizer):   token = tokenizer.token   data = token.data.upper()   if data == "WAVE":  self.ParseWave(tokenizer)   elif data == "QUIT":  self.ParseQuit(tokenizer)   elif data == "GIVEGOLD":  self.ParseGiveGold(tokenizer)   elif data == "SAYTO":  self.ParseSayTo(tokenizer)   else:  raise ParserError("Invalid command '" + data + "'")   #if token.type != TT_END_OF_INPUT:   #    self.type = "CT_NONE"   #    raise "Expected end of input!!! got " + token.type + " '" + token.data + "'"    def BuildFromInput(self, data):   tokenizer = Tokenizer(data)   token = tokenizer.GetNextToken()   if token.type != TT_BEGIN_COMMAND:  return   # skip the command token   token = tokenizer.GetNextToken()   if token.type == "TT_IDENTIFIER":  self.ParseCommand(tokenizer)   else:  raise ParserError("Expected IDENTIFIER, got " + token.type)done = Falsewhile not done:    data = raw_input(">")    command = Command()    try:   command.BuildFromInput(data)   if command.type == CT_NONE:  print "you said:", dataelse:  print command.type, command.data   if command.type == CT_QUIT:  done = True except ParserError, e:   print e [Edited by - smr on June 18, 2006 10:49:36 PM]