title: boost::tokenizer vs boost::split
body: <p>I am trying to parse a c++ string on every '^' character into a vector tokens. I have always used the boost::split method, but I am now writing performance critical code and would like to know which one gives better performance.</p>
  
  <p>For example:</p>
  
  <pre><code>string message = "A^B^C^D";
  vector&lt;string&gt; tokens;
  boost::split(tokens, message, boost::is_any_of("^"));
  </code></pre>
  
  <p>vs.</p>
  
  <pre><code>boost::char_separator&lt;char&gt; sep("^");
  boost::tokenizer&lt;boost::char_separator&lt;char&gt; &gt; tokens(text, sep);
  </code></pre>
  
  <p>Which one would give better performance and why?</p>