title: Parsing input into tokens using different delimiters in C++
body: <p>So I am trying to parse input data and break up the data into string tokens, the problem is, I sometimes need different delimiters on the same line and on different lines, all at the same time. Here is an example of what I need to parse:</p>
  
  <p>Input:</p>
  
  <blockquote>
  <pre><code>&lt;15&gt;
  algorithm [binary tree] analysis heap
  &lt;1&gt;
  [binary search tree] analysis
  complexity algorithm [2-3 tree]
  &lt;5&gt;
  tree [b+ tree] [Binary Tree]
  &lt;8&gt;
  graph clique Tree
  &lt;5&gt;
  tree [full binary tree]
  [complete binary tree]
  &lt;-1&gt;
  </code></pre>
  </blockquote>
  
  <p>So from the above input, I need to take parse the numbers inside angle brackets using a delimeter &lt;> which I have done and works in my code. Then I need to parse all the data on those lines using both the dilemeter " " for individual words, and "[]" as a delimeter for the words inside the brackets which need to have the spaces included.</p>
  
  <p>So this is currently what I have:</p>
  
  <pre><code>// create a file-reading object
  ifstream fin;
  fin.open("input.txt"); // open a file
  if (!fin.good())
  return 1; // exit if file not found
  
  // read each line of the file
  while (!fin.eof())
  {
  // read an entire line into memory
  char buf[maxChars];
  fin.getline(buf, maxChars);
  cout &lt;&lt; "The line I'm about to read is: "&lt;&lt;  buf &lt;&lt; endl;
  
  int n = 0; // a for-loop index
  
  // array to store memory addresses of the tokens in buf
  char* token[maxTokens] = {}; // initialize to 0
  
  // parse the line into &lt;&gt; delimited tokens first
  token[0] = strtok(buf, "&lt;&gt;"); // number tokens with &lt;&gt; delimeters
  if (token[0])
  {
    token[1] = strtok(token[0], "[]"); //break tokens by spaces
    for (n = 1; n &lt;20; n++)
    {
      token[n]=strtok(NULL, " "); //break tokens by brackets
      if (!token[n]) break; // no more tokens
    }
  }
  
  // process (print) the tokens
  for (int i = 0; i &lt; n; i++) // n = #of tokens
    cout &lt;&lt; "Token[" &lt;&lt; i &lt;&lt; "] = " &lt;&lt; token[i] &lt;&lt; endl;
  cout &lt;&lt; endl;
  }
  </code></pre>
  
  <p>and I get this kind of output:</p>
  
  <blockquote>
  <pre><code>The line I'm about to read is: &lt;15&gt;
  Token[0] = 15      
  </code></pre>
  </blockquote>
  
  <pre><code>The line I'm about to read is: algorithm [binary tree] analysis heap
  Token[0] = algorithm 
  Token[1] = binary tree
  Token[2] =  analysis heap
  
  The line I'm about to read is: &lt;1&gt;
  Token[0] = 1
  
  The line I'm about to read is: [binary search tree] analysis
  Token[0] = [binary search tree
  Token[1] =  analysis
  
  The line I'm about to read is: complexity algorithm [2-3 tree]
  Token[0] = complexity algorithm 
  Token[1] = 2-3 tree
  
  The line I'm about to read is: &lt;5&gt;
  Token[0] = 5
  
  The line I'm about to read is: tree [b+ tree] [Binary Tree]
  Token[0] = tree 
  Token[1] = b+ tree
  Token[2] =  
  Token[3] = Binary Tree
  
  The line I'm about to read is: &lt;8&gt;
  Token[0] = 8
  
  The line I'm about to read is: graph clique Tree
  Token[0] = graph clique Tree
  
  The line I'm about to read is: &lt;5&gt;
  Token[0] = 5
  
  The line I'm about to read is: tree [full binary tree]
  Token[0] = tree 
  Token[1] = full binary tree
  
  The line I'm about to read is: [complete binary tree]
  Token[0] = [complete binary tree
  
  The line I'm about to read is: &lt;-1&gt;
  Token[0] = -1
  </code></pre>
  
  <p>Basically I need it to look like:</p>
  
  <blockquote>
    <p>Token[0] = 15<br>
        Token[0] = algorithm<br>
        Token[1] = binary tree<br>
        Token[2] = analysis<br>
        Token[3] = heap<br>
    and etc....</p>
  </blockquote>