title: Removing duplicates from Boost::Tokenizer?
body: <p>I am trying to split a comma-separated string and then perform some action on each token, but ignoring duplicates, so sth. along the following lines:</p>
  
  <pre><code>int main(int, char**)
  {
     string text = "token, test   string";
  
    char_separator&lt;char&gt; sep(", ");
    tokenizer&lt; char_separator&lt;char&gt; &gt; tokens(text, sep);
    // remove duplicates from tokens?
    BOOST_FOREACH (const string&amp; t, tokens) {
      cout &lt;&lt; t &lt;&lt; "." &lt;&lt; endl;
    }
  }
  </code></pre>
  
  <p>Is there a way to do this on the boost::tokenizer?</p>
  
  <p>I know that I can solve this problem using boost::split and std::unique, but was wondering whether there is a way to achieve this with the tokenizer as well.</p>