<p>Java has a convenient split method:</p>

<pre><code>String str = "The quick brown fox";
String[] results = str.split(" ");
</code></pre>

<p>Is there an easy way to do this in C++?</p>

[{'answer_id': 53862, 'author': 'vzczc', 'author_id': 224, 'author_profile': 'https://Stackoverflow.com/users/224', 'pm_score': 5, 'selected': False, 'text': '<p>Here is a sample tokenizer class that might do what you want </p>\n\n<pre><code>//Header file\nclass Tokenizer \n{\n    public:\n        static const std::string DELIMITERS;\n        Tokenizer(const std::string&amp; str);\n        Tokenizer(const std::string&amp; str, const std::string&amp; delimiters);\n        bool NextToken();\n        bool NextToken(const std::string&amp; delimiters);\n        const std::string GetToken() const;\n        void Reset();\n    protected:\n        size_t m_offset;\n        const std::string m_string;\n        std::string m_token;\n        std::string m_delimiters;\n};\n\n//CPP file\nconst std::string Tokenizer::DELIMITERS(" \\t\\n\\r");\n\nTokenizer::Tokenizer(const std::string&amp; s) :\n    m_string(s), \n    m_offset(0), \n    m_delimiters(DELIMITERS) {}\n\nTokenizer::Tokenizer(const std::string&amp; s, const std::string&amp; delimiters) :\n    m_string(s), \n    m_offset(0), \n    m_delimiters(delimiters) {}\n\nbool Tokenizer::NextToken() \n{\n    return NextToken(m_delimiters);\n}\n\nbool Tokenizer::NextToken(const std::string&amp; delimiters) \n{\n    size_t i = m_string.find_first_not_of(delimiters, m_offset);\n    if (std::string::npos == i) \n    {\n        m_offset = m_string.length();\n        return false;\n    }\n\n    size_t j = m_string.find_first_of(delimiters, i);\n    if (std::string::npos == j) \n    {\n        m_token = m_string.substr(i);\n        m_offset = m_string.length();\n        return true;\n    }\n\n    m_token = m_string.substr(i, j - i);\n    m_offset = j;\n    return true;\n}\n</code></pre>\n\n<p>Example:</p>\n\n<pre><code>std::vector &lt;std::string&gt; v;\nTokenizer s("split this string", " ");\nwhile (s.NextToken())\n{\n    v.push_back(s.GetToken());\n}\n</code></pre>\n'}, {'answer_id': 53863, 'author': 'Konrad Rudolph', 'author_id': 1968, 'author_profile': 'https://Stackoverflow.com/users/1968', 'pm_score': 8, 'selected': True, 'text': '<p>C++ standard library algorithms are pretty universally based around iterators rather than concrete containers. Unfortunately this makes it hard to provide a Java-like <code>split</code> function in the C++ standard library, even though nobody argues that this would be convenient. But what would its return type be? <code>std::vector&lt;std::basic_string&lt;…&gt;&gt;</code>? Maybe, but then we’re forced to perform (potentially redundant and costly) allocations.</p>\n\n<p>Instead, C++ offers a plethora of ways to split strings based on arbitrarily complex delimiters, but none of them is encapsulated as nicely as in other languages. The numerous ways <a href="https://www.fluentcpp.com/2017/04/21/how-to-split-a-string-in-c/" rel="noreferrer">fill whole blog posts</a>.</p>\n\n<p>At its simplest, you could iterate using <a href="http://en.cppreference.com/w/cpp/string/basic_string/find" rel="noreferrer"><code>std::string::find</code></a> until you hit <code>std::string::npos</code>, and extract the contents using <a href="https://en.cppreference.com/w/cpp/string/basic_string/substr" rel="noreferrer"><code>std::string::substr</code></a>.</p>\n\n<p>A more fluid (and idiomatic, but basic) version for splitting on whitespace would use a <a href="https://en.cppreference.com/w/cpp/io/basic_istringstream" rel="noreferrer"><code>std::istringstream</code></a>:</p>\n\n<pre><code>auto iss = std::istringstream{"The quick brown fox"};\nauto str = std::string{};\n\nwhile (iss &gt;&gt; str) {\n    process(str);\n}\n</code></pre>\n\n<p>Using <a href="https://en.cppreference.com/w/cpp/iterator/istream_iterator" rel="noreferrer"><code>std::istream_iterator</code>s</a>, the contents of the string stream could also be copied into a vector using its iterator range constructor.</p>\n\n<p>Multiple libraries (such as <a href="https://www.boost.org/doc/libs/1_70_0/libs/tokenizer/doc/tokenizer.htm" rel="noreferrer">Boost.Tokenizer</a>) offer specific tokenisers.</p>\n\n<p>More advanced splitting require regular expressions. C++ provides the <a href="https://en.cppreference.com/w/cpp/regex/regex_token_iterator" rel="noreferrer"><code>std::regex_token_iterator</code></a> for this purpose in particular:</p>\n\n<pre><code>auto const str = "The quick brown fox"s;\nauto const re = std::regex{R"(\\s+)"};\nauto const vec = std::vector&lt;std::string&gt;(\n    std::sregex_token_iterator{begin(str), end(str), re, -1},\n    std::sregex_token_iterator{}\n);\n</code></pre>\n'}, {'answer_id': 53867, 'author': 'On Freund', 'author_id': 2150, 'author_profile': 'https://Stackoverflow.com/users/2150', 'pm_score': 2, 'selected': False, 'text': '<p>If you\'re willing to use C, you can use the <a href="http://www.cplusplus.com/reference/clibrary/cstring/strtok.html" rel="nofollow noreferrer">strtok</a> function. You should pay attention to multi-threading issues when using it.</p>\n'}, {'answer_id': 53878, 'author': 'Adam Pierce', 'author_id': 5324, 'author_profile': 'https://Stackoverflow.com/users/5324', 'pm_score': 7, 'selected': False, 'text': "<p>Here's a real simple one:</p>\n\n<pre><code>#include &lt;vector&gt;\n#include &lt;string&gt;\nusing namespace std;\n\nvector&lt;string&gt; split(const char *str, char c = ' ')\n{\n    vector&lt;string&gt; result;\n\n    do\n    {\n        const char *begin = str;\n\n        while(*str != c &amp;&amp; *str)\n            str++;\n\n        result.push_back(string(begin, str));\n    } while (0 != *str++);\n\n    return result;\n}\n</code></pre>\n"}, {'answer_id': 53915, 'author': 'Daren Thomas', 'author_id': 2260, 'author_profile': 'https://Stackoverflow.com/users/2260', 'pm_score': 2, 'selected': False, 'text': '<p>I thought that was what the <code>&gt;&gt;</code> operator on string streams was for:</p>\n\n<pre><code>string word; sin &gt;&gt; word;\n</code></pre>\n'}, {'answer_id': 53921, 'author': 'KeithB', 'author_id': 2298, 'author_profile': 'https://Stackoverflow.com/users/2298', 'pm_score': 6, 'selected': False, 'text': '<p>You can use streams, iterators, and the copy algorithm to do this fairly directly.</p>\n\n<pre><code>#include &lt;string&gt;\n#include &lt;vector&gt;\n#include &lt;iostream&gt;\n#include &lt;istream&gt;\n#include &lt;ostream&gt;\n#include &lt;iterator&gt;\n#include &lt;sstream&gt;\n#include &lt;algorithm&gt;\n\nint main()\n{\n  std::string str = "The quick brown fox";\n\n  // construct a stream from the string\n  std::stringstream strstr(str);\n\n  // use stream iterators to copy the stream to the vector as whitespace separated strings\n  std::istream_iterator&lt;std::string&gt; it(strstr);\n  std::istream_iterator&lt;std::string&gt; end;\n  std::vector&lt;std::string&gt; results(it, end);\n\n  // send the vector to stdout.\n  std::ostream_iterator&lt;std::string&gt; oit(std::cout);\n  std::copy(results.begin(), results.end(), oit);\n}\n</code></pre>\n'}, {'answer_id': 54048, 'author': 'Mark', 'author_id': 4405, 'author_profile': 'https://Stackoverflow.com/users/4405', 'pm_score': 7, 'selected': False, 'text': '<p>Use strtok.  In my opinion, there isn\'t a need to build a class around tokenizing unless strtok doesn\'t provide you with what you need.  It might not, but in 15+ years of writing various parsing code in C and C++, I\'ve always used strtok.  Here is an example</p>\n\n<pre><code>char myString[] = "The quick brown fox";\nchar *p = strtok(myString, " ");\nwhile (p) {\n    printf ("Token: %s\\n", p);\n    p = strtok(NULL, " ");\n}\n</code></pre>\n\n<p>A few caveats (which might not suit your needs).  The string is "destroyed" in the process, meaning that EOS characters are placed inline in the delimter spots.  Correct usage might require you to make a non-const version of the string.  You can also change the list of delimiters mid parse.</p>\n\n<p>In my own opinion, the above code is far simpler and easier to use than writing a separate class for it.  To me, this is one of those functions that the language provides and it does it well and cleanly.  It\'s simply a "C based" solution.  It\'s appropriate, it\'s easy, and you don\'t have to write a lot of extra code :-)</p>\n'}, {'answer_id': 55680, 'author': 'Ferruccio', 'author_id': 4086, 'author_profile': 'https://Stackoverflow.com/users/4086', 'pm_score': 8, 'selected': False, 'text': '<p>The <a href="http://www.boost.org/doc/libs/1_36_0/libs/tokenizer/index.html" rel="noreferrer">Boost tokenizer</a> class can make this sort of thing quite simple:</p>\n\n<pre><code>#include &lt;iostream&gt;\n#include &lt;string&gt;\n#include &lt;boost/foreach.hpp&gt;\n#include &lt;boost/tokenizer.hpp&gt;\n\nusing namespace std;\nusing namespace boost;\n\nint main(int, char**)\n{\n    string text = "token, test   string";\n\n    char_separator&lt;char&gt; sep(", ");\n    tokenizer&lt; char_separator&lt;char&gt; &gt; tokens(text, sep);\n    BOOST_FOREACH (const string&amp; t, tokens) {\n        cout &lt;&lt; t &lt;&lt; "." &lt;&lt; endl;\n    }\n}\n</code></pre>\n\n<p>Updated for C++11:</p>\n\n<pre><code>#include &lt;iostream&gt;\n#include &lt;string&gt;\n#include &lt;boost/tokenizer.hpp&gt;\n\nusing namespace std;\nusing namespace boost;\n\nint main(int, char**)\n{\n    string text = "token, test   string";\n\n    char_separator&lt;char&gt; sep(", ");\n    tokenizer&lt;char_separator&lt;char&gt;&gt; tokens(text, sep);\n    for (const auto&amp; t : tokens) {\n        cout &lt;&lt; t &lt;&lt; "." &lt;&lt; endl;\n    }\n}\n</code></pre>\n'}, {'answer_id': 59552, 'author': 'Raz', 'author_id': 5661, 'author_profile': 'https://Stackoverflow.com/users/5661', 'pm_score': 5, 'selected': False, 'text': '<p><a href="http://en.wikipedia.org/wiki/Boost_C%2B%2B_Libraries" rel="noreferrer">Boost</a> has a strong split function: <a href="http://www.boost.org/doc/libs/1_36_0/doc/html/boost/algorithm/split_id2965593.html" rel="noreferrer">boost::algorithm::split</a>.</p>\n\n<p>Sample program:</p>\n\n<pre><code>#include &lt;vector&gt;\n#include &lt;boost/algorithm/string.hpp&gt;\n\nint main() {\n    auto s = "a,b, c ,,e,f,";\n    std::vector&lt;std::string&gt; fields;\n    boost::split(fields, s, boost::is_any_of(","));\n    for (const auto&amp; field : fields)\n        std::cout &lt;&lt; "\\"" &lt;&lt; field &lt;&lt; "\\"\\n";\n    return 0;\n}\n</code></pre>\n\n<p>Output:</p>\n\n<pre><code>"a"\n"b"\n" c "\n""\n"e"\n"f"\n""\n</code></pre>\n'}, {'answer_id': 63946, 'author': 'jilles de wit', 'author_id': 7531, 'author_profile': 'https://Stackoverflow.com/users/7531', 'pm_score': 2, 'selected': False, 'text': '<p>For simple stuff I just use the following:</p>\n\n<pre><code>unsigned TokenizeString(const std::string&amp; i_source,\n                        const std::string&amp; i_seperators,\n                        bool i_discard_empty_tokens,\n                        std::vector&lt;std::string&gt;&amp; o_tokens)\n{\n    unsigned prev_pos = 0;\n    unsigned pos = 0;\n    unsigned number_of_tokens = 0;\n    o_tokens.clear();\n    pos = i_source.find_first_of(i_seperators, pos);\n    while (pos != std::string::npos)\n    {\n        std::string token = i_source.substr(prev_pos, pos - prev_pos);\n        if (!i_discard_empty_tokens || token != "")\n        {\n            o_tokens.push_back(i_source.substr(prev_pos, pos - prev_pos));\n            number_of_tokens++;\n        }\n\n        pos++;\n        prev_pos = pos;\n        pos = i_source.find_first_of(i_seperators, pos);\n    }\n\n    if (prev_pos &lt; i_source.length())\n    {\n        o_tokens.push_back(i_source.substr(prev_pos));\n        number_of_tokens++;\n    }\n\n    return number_of_tokens;\n}\n</code></pre>\n\n<p>Cowardly disclaimer: I write real-time data processing software where the data comes in through binary files, sockets, or some API call (I/O cards, camera\'s). I never use this function for something more complicated or time-critical than reading external configuration files on startup.</p>\n'}, {'answer_id': 325000, 'author': 'Mr.Ree', 'author_id': 37946, 'author_profile': 'https://Stackoverflow.com/users/37946', 'pm_score': 6, 'selected': False, 'text': '<p>No offense folks, but for such a simple problem, you are making things <strong><em>way</em></strong> too complicated. There are a lot of reasons to use <a href="http://en.wikipedia.org/wiki/Boost_C%2B%2B_Libraries" rel="nofollow noreferrer">Boost</a>. But for something this simple, it\'s like hitting a fly with a 20# sledge.</p>\n\n<pre><code>void\nsplit( vector&lt;string&gt; &amp; theStringVector,  /* Altered/returned value */\n       const  string  &amp; theString,\n       const  string  &amp; theDelimiter)\n{\n    UASSERT( theDelimiter.size(), &gt;, 0); // My own ASSERT macro.\n\n    size_t  start = 0, end = 0;\n\n    while ( end != string::npos)\n    {\n        end = theString.find( theDelimiter, start);\n\n        // If at end, use length=maxLength.  Else use length=end-start.\n        theStringVector.push_back( theString.substr( start,\n                       (end == string::npos) ? string::npos : end - start));\n\n        // If at end, use start=maxSize.  Else use start=end+delimiter.\n        start = (   ( end &gt; (string::npos - theDelimiter.size()) )\n                  ?  string::npos  :  end + theDelimiter.size());\n    }\n}\n</code></pre>\n\n<p>For example (for Doug\'s case),</p>\n\n<pre><code>#define SHOW(I,X)   cout &lt;&lt; "[" &lt;&lt; (I) &lt;&lt; "]\\t " # X " = \\"" &lt;&lt; (X) &lt;&lt; "\\"" &lt;&lt; endl\n\nint\nmain()\n{\n    vector&lt;string&gt; v;\n\n    split( v, "A:PEP:909:Inventory Item", ":" );\n\n    for (unsigned int i = 0;  i &lt; v.size();   i++)\n        SHOW( i, v[i] );\n}\n</code></pre>\n\n<p>And yes, we could have split() return a new vector rather than passing one in. It\'s trivial to wrap and overload. But depending on what I\'m doing, I often find it better to re-use pre-existing objects rather than always creating new ones. (Just as long as I don\'t forget to empty the vector in between!)</p>\n\n<p>Reference: <a href="http://en.cppreference.com/w/cpp/string/basic_string" rel="nofollow noreferrer">http://www.cplusplus.com/reference/string/string/</a>.</p>\n\n<p>(I was originally writing a response to Doug\'s question: <a href="https://stackoverflow.com/questions/324867/c-strings-modifying-and-extracting-based-on-separators">C++ Strings Modifying and Extracting based on Separators (closed)</a>. But since Martin York closed that question with a pointer over here... I\'ll just generalize my code.)</p>\n'}, {'answer_id': 325042, 'author': 'user35978', 'author_id': 35978, 'author_profile': 'https://Stackoverflow.com/users/35978', 'pm_score': 7, 'selected': False, 'text': '<p>Another quick way is to use <code>getline</code>. Something like:</p>\n\n<pre><code>stringstream ss("bla bla");\nstring s;\n\nwhile (getline(ss, s, \' \')) {\n cout &lt;&lt; s &lt;&lt; endl;\n}\n</code></pre>\n\n<p>If you want, you can make a simple <code>split()</code> method returning a <code>vector&lt;string&gt;</code>, which is \nreally useful. </p>\n'}, {'answer_id': 670441, 'author': 'Jim In Texas', 'author_id': 15079, 'author_profile': 'https://Stackoverflow.com/users/15079', 'pm_score': 3, 'selected': False, 'text': '<p>MFC/ATL has a very nice tokenizer.  From MSDN:</p>\n\n<pre><code>CAtlString str( "%First Second#Third" );\nCAtlString resToken;\nint curPos= 0;\n\nresToken= str.Tokenize("% #",curPos);\nwhile (resToken != "")\n{\n   printf("Resulting token: %s\\n", resToken);\n   resToken= str.Tokenize("% #",curPos);\n};\n\nOutput\n\nResulting Token: First\nResulting Token: Second\nResulting Token: Third\n</code></pre>\n'}, {'answer_id': 3408134, 'author': 'sivabudh', 'author_id': 65313, 'author_profile': 'https://Stackoverflow.com/users/65313', 'pm_score': 5, 'selected': False, 'text': '<p>I know you asked for a C++ solution, but you might consider this helpful:</p>\n\n<p><strong>Qt</strong></p>\n\n<pre><code>#include &lt;QString&gt;\n\n...\n\nQString str = "The quick brown fox"; \nQStringList results = str.split(" "); \n</code></pre>\n\n<p>The advantage over Boost in this example is that it\'s a direct one to one mapping to your post\'s code.</p>\n\n<p>See more at <a href="https://doc.qt.io/qt-5/qstring.html#split" rel="noreferrer">Qt documentation</a></p>\n'}, {'answer_id': 4489586, 'author': 'sohesado', 'author_id': 548600, 'author_profile': 'https://Stackoverflow.com/users/548600', 'pm_score': 3, 'selected': False, 'text': '<p>Check this example. It might help you..</p>\n\n<pre><code>#include &lt;iostream&gt;\n#include &lt;sstream&gt;\n\nusing namespace std;\n\nint main ()\n{\n    string tmps;\n    istringstream is ("the dellimiter is the space");\n    while (is.good ()) {\n        is &gt;&gt; tmps;\n        cout &lt;&lt; tmps &lt;&lt; "\\n";\n    }\n    return 0;\n}\n</code></pre>\n'}, {'answer_id': 5751134, 'author': 'Fawix', 'author_id': 706840, 'author_profile': 'https://Stackoverflow.com/users/706840', 'pm_score': 2, 'selected': False, 'text': '<p>You can simply use a <a href="http://www.tropicsoft.com/Components/RegularExpression/" rel="nofollow noreferrer">regular expression library</a> and solve that using regular expressions.</p>\n\n<p>Use expression (\\w+) and the variable in \\1 (or $1 depending on the library implementation of regular expressions).</p>\n'}, {'answer_id': 6011144, 'author': 'Angel Sinigersky', 'author_id': 754396, 'author_profile': 'https://Stackoverflow.com/users/754396', 'pm_score': 0, 'selected': False, 'text': '<p>If the maximum length of the input string to be tokenized is known, one can exploit this and implement a very fast version. I am sketching the basic idea below, which was inspired by both strtok() and the "suffix array"-data structure described Jon Bentley\'s "Programming Perls" 2nd edition, chapter 15. The C++ class in this case only gives some organization and convenience of use. The implementation shown can be easily extended for removing leading and trailing whitespace characters in the tokens.</p>\n\n<p>Basically one can replace the separator characters with string-terminating \'\\0\'-characters and set pointers to the tokens withing the modified string. In the extreme case when the string consists only of separators, one gets string-length plus 1 resulting empty tokens. It is practical to duplicate the string to be modified.</p>\n\n<p>Header file:</p>\n\n<pre><code>class TextLineSplitter\n{\npublic:\n\n    TextLineSplitter( const size_t max_line_len );\n\n    ~TextLineSplitter();\n\n    void            SplitLine( const char *line,\n                               const char sep_char = \',\',\n                             );\n\n    inline size_t   NumTokens( void ) const\n    {\n        return mNumTokens;\n    }\n\n    const char *    GetToken( const size_t token_idx ) const\n    {\n        assert( token_idx &lt; mNumTokens );\n        return mTokens[ token_idx ];\n    }\n\nprivate:\n    const size_t    mStorageSize;\n\n    char           *mBuff;\n    char          **mTokens;\n    size_t          mNumTokens;\n\n    inline void     ResetContent( void )\n    {\n        memset( mBuff, 0, mStorageSize );\n        // mark all items as empty:\n        memset( mTokens, 0, mStorageSize * sizeof( char* ) );\n        // reset counter for found items:\n        mNumTokens = 0L;\n    }\n};\n</code></pre>\n\n<p>Implementattion file:</p>\n\n<pre><code>TextLineSplitter::TextLineSplitter( const size_t max_line_len ):\n    mStorageSize ( max_line_len + 1L )\n{\n    // allocate memory\n    mBuff   = new char  [ mStorageSize ];\n    mTokens = new char* [ mStorageSize ];\n\n    ResetContent();\n}\n\nTextLineSplitter::~TextLineSplitter()\n{\n    delete [] mBuff;\n    delete [] mTokens;\n}\n\n\nvoid TextLineSplitter::SplitLine( const char *line,\n                                  const char sep_char   /* = \',\' */,\n                                )\n{\n    assert( sep_char != \'\\0\' );\n\n    ResetContent();\n    strncpy( mBuff, line, mMaxLineLen );\n\n    size_t idx       = 0L; // running index for characters\n\n    do\n    {\n        assert( idx &lt; mStorageSize );\n\n        const char chr = line[ idx ]; // retrieve current character\n\n        if( mTokens[ mNumTokens ] == NULL )\n        {\n            mTokens[ mNumTokens ] = &amp;mBuff[ idx ];\n        } // if\n\n        if( chr == sep_char || chr == \'\\0\' )\n        { // item or line finished\n            // overwrite separator with a 0-terminating character:\n            mBuff[ idx ] = \'\\0\';\n            // count-up items:\n            mNumTokens ++;\n        } // if\n\n    } while( line[ idx++ ] );\n}\n</code></pre>\n\n<p>A scenario of usage would be:</p>\n\n<pre><code>// create an instance capable of splitting strings up to 1000 chars long:\nTextLineSplitter spl( 1000 );\nspl.SplitLine( "Item1,,Item2,Item3" );\nfor( size_t i = 0; i &lt; spl.NumTokens(); i++ )\n{\n    printf( "%s\\n", spl.GetToken( i ) );\n}\n</code></pre>\n\n<p>output:</p>\n\n<pre><code>Item1\n\nItem2\nItem3\n</code></pre>\n'}, {'answer_id': 6922744, 'author': 'Arash', 'author_id': 551537, 'author_profile': 'https://Stackoverflow.com/users/551537', 'pm_score': 1, 'selected': False, 'text': '<p>you can take advantage of boost::make_find_iterator. Something similar to this:</p>\n\n<pre><code>template&lt;typename CH&gt;\ninline vector&lt; basic_string&lt;CH&gt; &gt; tokenize(\n    const basic_string&lt;CH&gt; &amp;Input,\n    const basic_string&lt;CH&gt; &amp;Delimiter,\n    bool remove_empty_token\n    ) {\n\n    typedef typename basic_string&lt;CH&gt;::const_iterator string_iterator_t;\n    typedef boost::find_iterator&lt; string_iterator_t &gt; string_find_iterator_t;\n\n    vector&lt; basic_string&lt;CH&gt; &gt; Result;\n    string_iterator_t it = Input.begin();\n    string_iterator_t it_end = Input.end();\n    for(string_find_iterator_t i = boost::make_find_iterator(Input, boost::first_finder(Delimiter, boost::is_equal()));\n        i != string_find_iterator_t();\n        ++i) {\n        if(remove_empty_token){\n            if(it != i-&gt;begin())\n                Result.push_back(basic_string&lt;CH&gt;(it,i-&gt;begin()));\n        }\n        else\n            Result.push_back(basic_string&lt;CH&gt;(it,i-&gt;begin()));\n        it = i-&gt;end();\n    }\n    if(it != it_end)\n        Result.push_back(basic_string&lt;CH&gt;(it,it_end));\n\n    return Result;\n}\n</code></pre>\n'}, {'answer_id': 8669587, 'author': 'dbr', 'author_id': 745, 'author_profile': 'https://Stackoverflow.com/users/745', 'pm_score': 4, 'selected': False, 'text': '<p><a href="https://github.com/imageworks/pystring" rel="noreferrer">pystring</a> is a small library which implements a bunch of Python\'s string functions, including the split method:</p>\n\n<pre><code>#include &lt;string&gt;\n#include &lt;vector&gt;\n#include "pystring.h"\n\nstd::vector&lt;std::string&gt; chunks;\npystring::split("this string", chunks);\n\n// also can specify a separator\npystring::split("this-string", chunks, "-");\n</code></pre>\n'}, {'answer_id': 11497058, 'author': 'jochenleidner', 'author_id': 1527677, 'author_profile': 'https://Stackoverflow.com/users/1527677', 'pm_score': 0, 'selected': False, 'text': '<p><a href="http://www.boost.org/doc/libs/1_50_0/libs/tokenizer/index.html" rel="nofollow"><code>boost::tokenizer</code></a> is your friend, but consider making your code portable with reference to internationalization (i18n) issues by using <code>wstring</code>/<code>wchar_t</code> instead of the legacy <code>string</code>/<code>char</code> types.</p>\n\n<pre><code>#include &lt;iostream&gt;\n#include &lt;boost/tokenizer.hpp&gt;\n#include &lt;string&gt;\n\nusing namespace std;\nusing namespace boost;\n\ntypedef tokenizer&lt;char_separator&lt;wchar_t&gt;,\n                  wstring::const_iterator, wstring&gt; Tok;\n\nint main()\n{\n  wstring s;\n  while (getline(wcin, s)) {\n    char_separator&lt;wchar_t&gt; sep(L" "); // list of separator characters\n    Tok tok(s, sep);\n    for (Tok::iterator beg = tok.begin(); beg != tok.end(); ++beg) {\n      wcout &lt;&lt; *beg &lt;&lt; L"\\t"; // output (or store in vector)\n    }\n    wcout &lt;&lt; L"\\n";\n  }\n  return 0;\n}\n</code></pre>\n'}, {'answer_id': 11753247, 'author': 'David919', 'author_id': 1567655, 'author_profile': 'https://Stackoverflow.com/users/1567655', 'pm_score': 2, 'selected': False, 'text': "<p>Many overly complicated suggestions here.  Try this simple std::string solution:</p>\n\n<pre><code>using namespace std;\n\nstring someText = ...\n\nstring::size_type tokenOff = 0, sepOff = tokenOff;\nwhile (sepOff != string::npos)\n{\n    sepOff = someText.find(' ', sepOff);\n    string::size_type tokenLen = (sepOff == string::npos) ? sepOff : sepOff++ - tokenOff;\n    string token = someText.substr(tokenOff, tokenLen);\n    if (!token.empty())\n        /* do something with token */;\n    tokenOff = sepOff;\n}\n</code></pre>\n"}, {'answer_id': 13089627, 'author': 'Darren Smith', 'author_id': 1776419, 'author_profile': 'https://Stackoverflow.com/users/1776419', 'pm_score': 2, 'selected': False, 'text': '<p>Here\'s an approach that allows you control over whether empty tokens are included (like strsep) or excluded (like strtok). </p>\n\n<pre class="lang-c prettyprint-override"><code>#include &lt;string.h&gt; // for strchr and strlen\n\n/*\n * want_empty_tokens==true  : include empty tokens, like strsep()\n * want_empty_tokens==false : exclude empty tokens, like strtok()\n */\nstd::vector&lt;std::string&gt; tokenize(const char* src,\n                                  char delim,\n                                  bool want_empty_tokens)\n{\n  std::vector&lt;std::string&gt; tokens;\n\n  if (src and *src != \'\\0\') // defensive\n    while( true )  {\n      const char* d = strchr(src, delim);\n      size_t len = (d)? d-src : strlen(src);\n\n      if (len or want_empty_tokens)\n        tokens.push_back( std::string(src, len) ); // capture token\n\n      if (d) src += len+1; else break;\n    }\n\n  return tokens;\n}\n</code></pre>\n'}, {'answer_id': 16635277, 'author': 'Karthik', 'author_id': 2398970, 'author_profile': 'https://Stackoverflow.com/users/2398970', 'pm_score': -1, 'selected': False, 'text': "<p>This a simple loop to tokenise with only standard library files</p>\n\n<pre><code>#include &lt;iostream.h&gt;\n#include &lt;stdio.h&gt;\n#include &lt;string.h&gt;\n#include &lt;math.h&gt;\n#include &lt;conio.h&gt;\nclass word\n    {\n     public:\n     char w[20];\n     word()\n      {\n        for(int j=0;j&lt;=20;j++)\n        {w[j]='\\0';\n      }\n   }\n\n\n\n};\n\nvoid main()\n  {\n    int i=1,n=0,j=0,k=0,m=1;\n    char input[100];\n    word ww[100];\n    gets(input);\n\n    n=strlen(input);\n\n\n    for(i=0;i&lt;=m;i++)\n      {\n        if(context[i]!=' ')\n         {\n            ww[k].w[j]=context[i];\n            j++;\n\n         }\n         else\n        {\n         k++;\n         j=0;\n         m++;\n        }\n\n   }\n }\n</code></pre>\n"}, {'answer_id': 20590157, 'author': 'vsoftco', 'author_id': 3093378, 'author_profile': 'https://Stackoverflow.com/users/3093378', 'pm_score': 0, 'selected': False, 'text': '<p>Simple C++ code (standard C++98), accepts multiple delimiters (specified in a std::string), uses only vectors, strings and iterators.</p>\n\n<pre><code>#include &lt;iostream&gt;\n#include &lt;vector&gt;\n#include &lt;string&gt;\n#include &lt;stdexcept&gt; \n\nstd::vector&lt;std::string&gt; \nsplit(const std::string&amp; str, const std::string&amp; delim){\n    std::vector&lt;std::string&gt; result;\n    if (str.empty())\n        throw std::runtime_error("Can not tokenize an empty string!");\n    std::string::const_iterator begin, str_it;\n    begin = str_it = str.begin(); \n    do {\n        while (delim.find(*str_it) == std::string::npos &amp;&amp; str_it != str.end())\n            str_it++; // find the position of the first delimiter in str\n        std::string token = std::string(begin, str_it); // grab the token\n        if (!token.empty()) // empty token only when str starts with a delimiter\n            result.push_back(token); // push the token into a vector&lt;string&gt;\n        while (delim.find(*str_it) != std::string::npos &amp;&amp; str_it != str.end())\n            str_it++; // ignore the additional consecutive delimiters\n        begin = str_it; // process the remaining tokens\n        } while (str_it != str.end());\n    return result;\n}\n\nint main() {\n    std::string test_string = ".this is.a.../.simple;;test;;;END";\n    std::string delim = "; ./"; // string containing the delimiters\n    std::vector&lt;std::string&gt; tokens = split(test_string, delim);           \n    for (std::vector&lt;std::string&gt;::const_iterator it = tokens.begin(); \n        it != tokens.end(); it++)\n            std::cout &lt;&lt; *it &lt;&lt; std::endl;\n}\n</code></pre>\n'}, {'answer_id': 20981311, 'author': 'DannyK', 'author_id': 969968, 'author_profile': 'https://Stackoverflow.com/users/969968', 'pm_score': 4, 'selected': False, 'text': '<p>I posted this answer for similar question.<br>\nDon\'t reinvent the wheel.  I\'ve used a number of libraries and the fastest and most flexible I have come across is:  <a href="http://www.partow.net/programming/strtk/" rel="nofollow noreferrer">C++ String Toolkit Library</a>.  </p>\n\n<p>Here is an example of how to use it that I\'ve posted else where on the stackoverflow.</p>\n\n<pre><code>#include &lt;iostream&gt;\n#include &lt;vector&gt;\n#include &lt;string&gt;\n#include &lt;strtk.hpp&gt;\n\nconst char *whitespace  = " \\t\\r\\n\\f";\nconst char *whitespace_and_punctuation  = " \\t\\r\\n\\f;,=";\n\nint main()\n{\n    {   // normal parsing of a string into a vector of strings\n       std::string s("Somewhere down the road");\n       std::vector&lt;std::string&gt; result;\n       if( strtk::parse( s, whitespace, result ) )\n       {\n           for(size_t i = 0; i &lt; result.size(); ++i )\n            std::cout &lt;&lt; result[i] &lt;&lt; std::endl;\n       }\n    }\n\n    {  // parsing a string into a vector of floats with other separators\n       // besides spaces\n\n       std::string s("3.0, 3.14; 4.0");\n       std::vector&lt;float&gt; values;\n       if( strtk::parse( s, whitespace_and_punctuation, values ) )\n       {\n           for(size_t i = 0; i &lt; values.size(); ++i )\n            std::cout &lt;&lt; values[i] &lt;&lt; std::endl;\n       }\n    }\n\n    {  // parsing a string into specific variables\n\n       std::string s("angle = 45; radius = 9.9");\n       std::string w1, w2;\n       float v1, v2;\n       if( strtk::parse( s, whitespace_and_punctuation, w1, v1, w2, v2) )\n       {\n           std::cout &lt;&lt; "word " &lt;&lt; w1 &lt;&lt; ", value " &lt;&lt; v1 &lt;&lt; std::endl;\n           std::cout &lt;&lt; "word " &lt;&lt; w2 &lt;&lt; ", value " &lt;&lt; v2 &lt;&lt; std::endl;\n       }\n    }\n\n    return 0;\n}\n</code></pre>\n'}, {'answer_id': 22006543, 'author': 'Murphy78', 'author_id': 3331297, 'author_profile': 'https://Stackoverflow.com/users/3331297', 'pm_score': 0, 'selected': False, 'text': '<pre><code>/// split a string into multiple sub strings, based on a separator string\n/// for example, if separator="::",\n///\n/// s = "abc" -&gt; "abc"\n///\n/// s = "abc::def xy::st:" -&gt; "abc", "def xy" and "st:",\n///\n/// s = "::abc::" -&gt; "abc"\n///\n/// s = "::" -&gt; NO sub strings found\n///\n/// s = "" -&gt; NO sub strings found\n///\n/// then append the sub-strings to the end of the vector v.\n/// \n/// the idea comes from the findUrls() function of "Accelerated C++", chapt7,\n/// findurls.cpp\n///\nvoid split(const string&amp; s, const string&amp; sep, vector&lt;string&gt;&amp; v)\n{\n    typedef string::const_iterator iter;\n    iter b = s.begin(), e = s.end(), i;\n    iter sep_b = sep.begin(), sep_e = sep.end();\n\n    // search through s\n    while (b != e){\n        i = search(b, e, sep_b, sep_e);\n\n        // no more separator found\n        if (i == e){\n            // it\'s not an empty string\n            if (b != e)\n                v.push_back(string(b, e));\n            break;\n        }\n        else if (i == b){\n            // the separator is found and right at the beginning\n            // in this case, we need to move on and search for the\n            // next separator\n            b = i + sep.length();\n        }\n        else{\n            // found the separator\n            v.push_back(string(b, i));\n            b = i;\n        }\n    }\n}\n</code></pre>\n\n<p>The boost library is good, but they are not always available. Doing this sort of things by hand is also a good brain exercise. Here we just use the std::search() algorithm from the STL, see the above code.</p>\n'}, {'answer_id': 22460450, 'author': 'robcsi', 'author_id': 3257292, 'author_profile': 'https://Stackoverflow.com/users/3257292', 'pm_score': 0, 'selected': False, 'text': '<p>I\'ve been searching for a way to split a string by a separator of any length, so I started writing it from scratch, as existing solutions didn\'t suit me.</p>\n\n<p>Here is my little algorithm, using only STL:</p>\n\n<pre><code>//use like this\n//std::vector&lt;std::wstring&gt; vec = Split&lt;std::wstring&gt; (L"Hello##world##!", L"##");\n\ntemplate &lt;typename valueType&gt;\nstatic std::vector &lt;valueType&gt; Split (valueType text, const valueType&amp; delimiter)\n{\n    std::vector &lt;valueType&gt; tokens;\n    size_t pos = 0;\n    valueType token;\n\n    while ((pos = text.find(delimiter)) != valueType::npos) \n    {\n        token = text.substr(0, pos);\n        tokens.push_back (token);\n        text.erase(0, pos + delimiter.length());\n    }\n    tokens.push_back (text);\n\n    return tokens;\n}\n</code></pre>\n\n<p>It can be used with separator of any length and form, as far as I\'ve tested. Instantiate with either string or wstring type.</p>\n\n<p>All the algorithm does is it searches for the delimiter, gets the part of the string that is up to the delimiter, deletes the delimiter and searches again until it finds it no more.</p>\n\n<p>Hope it helps.</p>\n'}, {'answer_id': 24971386, 'author': 'odinthenerd', 'author_id': 893819, 'author_profile': 'https://Stackoverflow.com/users/893819', 'pm_score': 2, 'selected': False, 'text': '<p>Seems odd to me that with all us speed conscious nerds here on SO no one has presented a version that uses a compile time generated look up table for the delimiter (example implementation further down). Using a look up table and iterators should beat std::regex in efficiency, if you don\'t need to beat regex, just use it, its standard as of C++11 and super flexible.</p>\n\n<p>Some have suggested regex already but for the noobs here is a packaged example that should do exactly what the OP expects:</p>\n\n<pre><code>std::vector&lt;std::string&gt; split(std::string::const_iterator it, std::string::const_iterator end, std::regex e = std::regex{"\\\\w+"}){\n    std::smatch m{};\n    std::vector&lt;std::string&gt; ret{};\n    while (std::regex_search (it,end,m,e)) {\n        ret.emplace_back(m.str());              \n        std::advance(it, m.position() + m.length()); //next start position = match position + match length\n    }\n    return ret;\n}\nstd::vector&lt;std::string&gt; split(const std::string &amp;s, std::regex e = std::regex{"\\\\w+"}){  //comfort version calls flexible version\n    return split(s.cbegin(), s.cend(), std::move(e));\n}\nint main ()\n{\n    std::string str {"Some people, excluding those present, have been compile time constants - since puberty."};\n    auto v = split(str);\n    for(const auto&amp;s:v){\n        std::cout &lt;&lt; s &lt;&lt; std::endl;\n    }\n    std::cout &lt;&lt; "crazy version:" &lt;&lt; std::endl;\n    v = split(str, std::regex{"[^e]+"});  //using e as delim shows flexibility\n    for(const auto&amp;s:v){\n        std::cout &lt;&lt; s &lt;&lt; std::endl;\n    }\n    return 0;\n}\n</code></pre>\n\n<p>If we need to be faster and accept the constraint that all chars must be 8 bits we can make a look up table at compile time using metaprogramming:</p>\n\n<pre><code>template&lt;bool...&gt; struct BoolSequence{};        //just here to hold bools\ntemplate&lt;char...&gt; struct CharSequence{};        //just here to hold chars\ntemplate&lt;typename T, char C&gt; struct Contains;   //generic\ntemplate&lt;char First, char... Cs, char Match&gt;    //not first specialization\nstruct Contains&lt;CharSequence&lt;First, Cs...&gt;,Match&gt; :\n    Contains&lt;CharSequence&lt;Cs...&gt;, Match&gt;{};     //strip first and increase index\ntemplate&lt;char First, char... Cs&gt;                //is first specialization\nstruct Contains&lt;CharSequence&lt;First, Cs...&gt;,First&gt;: std::true_type {}; \ntemplate&lt;char Match&gt;                            //not found specialization\nstruct Contains&lt;CharSequence&lt;&gt;,Match&gt;: std::false_type{};\n\ntemplate&lt;int I, typename T, typename U&gt; \nstruct MakeSequence;                            //generic\ntemplate&lt;int I, bool... Bs, typename U&gt; \nstruct MakeSequence&lt;I,BoolSequence&lt;Bs...&gt;, U&gt;:  //not last\n    MakeSequence&lt;I-1, BoolSequence&lt;Contains&lt;U,I-1&gt;::value,Bs...&gt;, U&gt;{};\ntemplate&lt;bool... Bs, typename U&gt; \nstruct MakeSequence&lt;0,BoolSequence&lt;Bs...&gt;,U&gt;{   //last  \n    using Type = BoolSequence&lt;Bs...&gt;;\n};\ntemplate&lt;typename T&gt; struct BoolASCIITable;\ntemplate&lt;bool... Bs&gt; struct BoolASCIITable&lt;BoolSequence&lt;Bs...&gt;&gt;{\n    /* could be made constexpr but not yet supported by MSVC */\n    static bool isDelim(const char c){\n        static const bool table[256] = {Bs...};\n        return table[static_cast&lt;int&gt;(c)];\n    }   \n};\nusing Delims = CharSequence&lt;\'.\',\',\',\' \',\':\',\'\\n\'&gt;;  //list your custom delimiters here\nusing Table = BoolASCIITable&lt;typename MakeSequence&lt;256,BoolSequence&lt;&gt;,Delims&gt;::Type&gt;;\n</code></pre>\n\n<p>With that in place making a <code>getNextToken</code> function is easy:</p>\n\n<pre><code>template&lt;typename T_It&gt;\nstd::pair&lt;T_It,T_It&gt; getNextToken(T_It begin,T_It end){\n    begin = std::find_if(begin,end,std::not1(Table{})); //find first non delim or end\n    auto second = std::find_if(begin,end,Table{});      //find first delim or end\n    return std::make_pair(begin,second);\n}\n</code></pre>\n\n<p>Using it is also easy:</p>\n\n<pre><code>int main() {\n    std::string s{"Some people, excluding those present, have been compile time constants - since puberty."};\n    auto it = std::begin(s);\n    auto end = std::end(s);\n    while(it != std::end(s)){\n        auto token = getNextToken(it,end);\n        std::cout &lt;&lt; std::string(token.first,token.second) &lt;&lt; std::endl;\n        it = token.second;\n    }\n    return 0;\n}\n</code></pre>\n\n<p>Here is a live example: <a href="http://ideone.com/GKtkLQ" rel="nofollow">http://ideone.com/GKtkLQ</a> </p>\n'}, {'answer_id': 27468529, 'author': 'w.b', 'author_id': 2720372, 'author_profile': 'https://Stackoverflow.com/users/2720372', 'pm_score': 6, 'selected': False, 'text': '<p>A solution using <code>regex_token_iterator</code>s:</p>\n\n<pre><code>#include &lt;iostream&gt;\n#include &lt;regex&gt;\n#include &lt;string&gt;\n\nusing namespace std;\n\nint main()\n{\n    string str("The quick brown fox");\n\n    regex reg("\\\\s+");\n\n    sregex_token_iterator iter(str.begin(), str.end(), reg, -1);\n    sregex_token_iterator end;\n\n    vector&lt;string&gt; vec(iter, end);\n\n    for (auto a : vec)\n    {\n        cout &lt;&lt; a &lt;&lt; endl;\n    }\n}\n</code></pre>\n'}, {'answer_id': 27969097, 'author': 'CATspellsDOG', 'author_id': 4308970, 'author_profile': 'https://Stackoverflow.com/users/4308970', 'pm_score': 0, 'selected': False, 'text': '<p>I made a lexer/tokenizer before with the use of only standard libraries. Here\'s the code:</p>\n\n<pre><code>#include &lt;iostream&gt;\n#include &lt;string&gt;\n#include &lt;vector&gt;\n#include &lt;sstream&gt;\n\nusing namespace std;\n\nstring seps(string&amp; s) {\n    if (!s.size()) return "";\n    stringstream ss;\n    ss &lt;&lt; s[0];\n    for (int i = 1; i &lt; s.size(); i++) {\n        ss &lt;&lt; \'|\' &lt;&lt; s[i];\n    }\n    return ss.str();\n}\n\nvoid Tokenize(string&amp; str, vector&lt;string&gt;&amp; tokens, const string&amp; delimiters = " ")\n{\n    seps(str);\n\n    // Skip delimiters at beginning.\n    string::size_type lastPos = str.find_first_not_of(delimiters, 0);\n    // Find first "non-delimiter".\n    string::size_type pos = str.find_first_of(delimiters, lastPos);\n\n    while (string::npos != pos || string::npos != lastPos)\n    {\n        // Found a token, add it to the vector.\n        tokens.push_back(str.substr(lastPos, pos - lastPos));\n        // Skip delimiters.  Note the "not_of"\n        lastPos = str.find_first_not_of(delimiters, pos);\n        // Find next "non-delimiter"\n        pos = str.find_first_of(delimiters, lastPos);\n    }\n}\n\nint main(int argc, char *argv[])\n{\n    vector&lt;string&gt; t;\n    string s = "Tokens for everyone!";\n\n    Tokenize(s, t, "|");\n\n    for (auto c : t)\n        cout &lt;&lt; c &lt;&lt; endl;\n\n    system("pause");\n\n    return 0;\n}\n</code></pre>\n'}, {'answer_id': 28788127, 'author': 'Parham', 'author_id': 1357387, 'author_profile': 'https://Stackoverflow.com/users/1357387', 'pm_score': 5, 'selected': False, 'text': '<p>This is a simple STL-only solution (~5 lines!) using <code>std::find</code> and <code>std::find_first_not_of</code> that handles repetitions of the delimiter (like spaces or periods for instance), as well leading and trailing delimiters:</p>\n\n<pre><code>#include &lt;string&gt;\n#include &lt;vector&gt;\n\nvoid tokenize(std::string str, std::vector&lt;string&gt; &amp;token_v){\n    size_t start = str.find_first_not_of(DELIMITER), end=start;\n\n    while (start != std::string::npos){\n        // Find next occurence of delimiter\n        end = str.find(DELIMITER, start);\n        // Push back the token found into vector\n        token_v.push_back(str.substr(start, end-start));\n        // Skip all occurences of the delimiter to find new start\n        start = str.find_first_not_of(DELIMITER, end);\n    }\n}\n</code></pre>\n\n<p>Try it out <a href="http://coliru.stacked-crooked.com/a/652f29c0500cf195"><em>live</em></a>!</p>\n'}, {'answer_id': 38595708, 'author': 'Jonathan Mee', 'author_id': 2642059, 'author_profile': 'https://Stackoverflow.com/users/2642059', 'pm_score': 3, 'selected': False, 'text': '<p><a href="https://stackoverflow.com/a/53878/2642059">Adam Pierce\'s answer</a> provides an hand-spun tokenizer taking in a <code>const char*</code>. It\'s a bit more problematic to do with iterators because <a href="https://stackoverflow.com/q/37209725/2642059">incrementing a <code>string</code>\'s end iterator is undefined</a>. That said, given <code>string str{ "The quick brown fox" }</code> we can certainly accomplish this:</p>\n\n<pre><code>auto start = find(cbegin(str), cend(str), \' \');\nvector&lt;string&gt; tokens{ string(cbegin(str), start) };\n\nwhile (start != cend(str)) {\n    const auto finish = find(++start, cend(str), \' \');\n\n    tokens.push_back(string(start, finish));\n    start = finish;\n}\n</code></pre>\n\n<p><a href="http://ideone.com/8bGCbg" rel="noreferrer"><kbd>Live Example</kbd></a></p>\n\n<hr>\n\n<p>If you\'re looking to abstract complexity by using standard functionality, as <a href="https://stackoverflow.com/a/53867/2642059">On Freund suggests</a> <a href="http://en.cppreference.com/w/cpp/string/byte/strtok" rel="noreferrer"><code>strtok</code></a> is a simple option: </p>\n\n<pre><code>vector&lt;string&gt; tokens;\n\nfor (auto i = strtok(data(str), " "); i != nullptr; i = strtok(nullptr, " ")) tokens.push_back(i);\n</code></pre>\n\n<p>If you don\'t have access to C++17 you\'ll need to substitute <code>data(str)</code> as in this example: <a href="http://ideone.com/8kAGoa" rel="noreferrer">http://ideone.com/8kAGoa</a></p>\n\n<p>Though not demonstrated in the example, <code>strtok</code> need not use the same delimiter for each token. Along with this advantage though, there are several drawbacks:</p>\n\n<ol>\n<li><code>strtok</code> cannot be used on multiple <code>strings</code> at the same time: Either a <code>nullptr</code> must be passed to continue tokenizing the current <code>string</code> or a new <code>char*</code> to tokenize must be passed (there are some non-standard implementations which do support this however, such as: <a href="https://msdn.microsoft.com/en-us/library/ftsafwz3.aspx" rel="noreferrer"><code>strtok_s</code></a>)</li>\n<li>For the same reason <code>strtok</code> cannot be used on multiple threads simultaneously (this may however be implementation defined, for example: <a href="https://msdn.microsoft.com/en-us/library/2c8d19sb.aspx#Anchor_3" rel="noreferrer">Visual Studio\'s implementation is thread safe</a>)</li>\n<li>Calling <code>strtok</code> modifies the <code>string</code> it is operating on, so it cannot be used on <code>const string</code>s, <code>const char*</code>s, or literal strings, to tokenize any of these with <code>strtok</code> or to operate on a <code>string</code> who\'s contents need to be preserved, <code>str</code> would have to be copied, then the copy could be operated on</li>\n</ol>\n\n<hr>\n\n<p><a href="/questions/tagged/c%2b%2b20" class="post-tag" title="show questions tagged &#39;c++20&#39;" rel="tag">c++20</a> provides us with <code>split_view</code> to tokenize strings, in a non-destructive manner: <a href="https://topanswers.xyz/cplusplus?q=749#a874" rel="noreferrer">https://topanswers.xyz/cplusplus?q=749#a874</a></p>\n\n<hr>\n\n<p>The previous methods cannot generate a tokenized <code>vector</code> in-place, meaning without abstracting them into a helper function they cannot initialize <code>const vector&lt;string&gt; tokens</code>. That functionality <em>and</em> the ability to accept <em>any</em> white-space delimiter can be harnessed using an <a href="http://en.cppreference.com/w/cpp/iterator/istream_iterator" rel="noreferrer"><code>istream_iterator</code></a>. For example given: <code>const string str{ "The  quick \\tbrown \\nfox" }</code> we can do this:</p>\n\n<pre><code>istringstream is{ str };\nconst vector&lt;string&gt; tokens{ istream_iterator&lt;string&gt;(is), istream_iterator&lt;string&gt;() };\n</code></pre>\n\n<p><a href="http://ideone.com/gWmfV9" rel="noreferrer"><kbd>Live Example</kbd></a></p>\n\n<p>The required construction of an <code>istringstream</code> for this option has far greater cost than the previous 2 options, however this cost is typically hidden in the expense of <code>string</code> allocation.</p>\n\n<hr>\n\n<p>If none of the above options are flexable enough for your tokenization needs, the most flexible option is using a <a href="http://en.cppreference.com/w/cpp/regex/regex_token_iterator" rel="noreferrer"><code>regex_token_iterator</code></a> of course with this flexibility comes greater expense, but again this is likely hidden in the <code>string</code> allocation cost. Say for example we want to tokenize based on non-escaped commas, also eating white-space, given the following input: <code>const string str{ "The ,qu\\\\,ick ,\\tbrown, fox" }</code> we can do this:</p>\n\n<pre><code>const regex re{ "\\\\s*((?:[^\\\\\\\\,]|\\\\\\\\.)*?)\\\\s*(?:,|$)" };\nconst vector&lt;string&gt; tokens{ sregex_token_iterator(cbegin(str), cend(str), re, 1), sregex_token_iterator() };\n</code></pre>\n\n<p><a href="http://ideone.com/q58zoX" rel="noreferrer"><kbd>Live Example</kbd></a></p>\n'}, {'answer_id': 50247503, 'author': 'NutCracker', 'author_id': 5517378, 'author_profile': 'https://Stackoverflow.com/users/5517378', 'pm_score': 2, 'selected': False, 'text': '<p>I know this question is already answered but I want to contribute. Maybe my solution is a bit simple but this is what I came up with:</p>\n\n<pre><code>vector&lt;string&gt; get_words(string const&amp; text, string const&amp; separator)\n{\n    vector&lt;string&gt; result;\n    string tmp = text;\n\n    size_t first_pos = 0;\n    size_t second_pos = tmp.find(separator);\n\n    while (second_pos != string::npos)\n    {\n        if (first_pos != second_pos)\n        {\n            string word = tmp.substr(first_pos, second_pos - first_pos);\n            result.push_back(word);\n        }\n        tmp = tmp.substr(second_pos + separator.length());\n        second_pos = tmp.find(separator);\n    }\n\n    result.push_back(tmp);\n\n    return result;\n}\n</code></pre>\n\n<p>Please comment if there is a better approach to something in my code or if something is wrong.</p>\n\n<p><strong>UPDATE:</strong> added generic separator</p>\n'}, {'answer_id': 52436170, 'author': 'kayleeFrye_onDeck', 'author_id': 3543437, 'author_profile': 'https://Stackoverflow.com/users/3543437', 'pm_score': 1, 'selected': False, 'text': '<p>Here\'s my Swiss® Army Knife of string-tokenizers for splitting up strings by whitespace, accounting for single and double-quote wrapped strings as well as stripping those characters from the results. I used RegexBuddy 4.x to generate <em>most</em> of the code-snippet, but I added custom handling for stripping quotes and a few other things.</p>\n\n<pre><code>#include &lt;string&gt;\n#include &lt;locale&gt;\n#include &lt;regex&gt;\n\nstd::vector&lt;std::wstring&gt; tokenize_string(std::wstring string_to_tokenize) {\n    std::vector&lt;std::wstring&gt; tokens;\n\n    std::wregex re(LR"(("[^"]*"|\'[^\']*\'|[^"\' ]+))", std::regex_constants::collate);\n\n    std::wsregex_iterator next( string_to_tokenize.begin(),\n                                string_to_tokenize.end(),\n                                re,\n                                std::regex_constants::match_not_null );\n\n    std::wsregex_iterator end;\n    const wchar_t single_quote = L\'\\\'\';\n    const wchar_t double_quote = L\'\\"\';\n    while ( next != end ) {\n        std::wsmatch match = *next;\n        const std::wstring token = match.str( 0 );\n        next++;\n\n        if (token.length() &gt; 2 &amp;&amp; (token.front() == double_quote || token.front() == single_quote))\n            tokens.emplace_back( std::wstring(token.begin()+1, token.begin()+token.length()-1) );\n        else\n            tokens.emplace_back(token);\n    }\n    return tokens;\n}\n</code></pre>\n'}, {'answer_id': 63431488, 'author': 'einpoklum', 'author_id': 1593077, 'author_profile': 'https://Stackoverflow.com/users/1593077', 'pm_score': 3, 'selected': False, 'text': '<p>If you\'re using C++ ranges - the full <a href="https://github.com/ericniebler/range-v3/" rel="nofollow noreferrer">ranges-v3</a> library, not the limited functionality accepted into C++20 - you could do it this way:</p>\n<pre><code>auto results = str | ranges::views::tokenize(&quot; &quot;,1);\n</code></pre>\n<p>... and this is lazily-evaluated. You can alternatively set a vector to this range:</p>\n<pre><code>auto results = str | ranges::views::tokenize(&quot; &quot;,1) | ranges::to&lt;std::vector&gt;();\n</code></pre>\n<p>this will take O(m) space and O(n) time if <code>str</code> has n characters making up m words.</p>\n<p>See also the library\'s own tokenization example, <a href="https://github.com/ericniebler/range-v3/blob/5b492996048d1baad98b50ec074e783eb6086627/test/view/tokenize.cpp" rel="nofollow noreferrer">here</a>.</p>\n'}, {'answer_id': 71564188, 'author': 'Tanzer', 'author_id': 3976739, 'author_profile': 'https://Stackoverflow.com/users/3976739', 'pm_score': 1, 'selected': False, 'text': '<p>I wrote a simplified version (and maybe a little bit efficient) of <a href="https://stackoverflow.com/a/50247503/3976739">https://stackoverflow.com/a/50247503/3976739</a> for my own use. I hope it would help.</p>\n<pre><code>void StrTokenizer(string&amp; source, const char* delimiter, vector&lt;string&gt;&amp; Tokens)\n{   \n   size_t new_index = 0;\n   size_t old_index = 0;\n\n   while (new_index != std::string::npos)   \n   {\n      new_index = source.find(delimiter, old_index);\n      Tokens.emplace_back(source.substr(old_index, new_index-old_index));\n\n      if (new_index != std::string::npos)\n          old_index = ++new_index;\n   }\n}\n</code></pre>\n'}, {'answer_id': 74347767, 'author': 'leanid.chaika', 'author_id': 955508, 'author_profile': 'https://Stackoverflow.com/users/955508', 'pm_score': 0, 'selected': False, 'text': "<p>I just read all the answers and can't find solution with next preconditions:</p>\n<ol>\n<li>no dynamic memory allocations</li>\n<li>no use of boost</li>\n<li>no use of regex</li>\n<li>c++17 standard only</li>\n</ol>\n<p>So here is my solution</p>\n<pre><code>#include &lt;iomanip&gt;\n#include &lt;iostream&gt;\n#include &lt;iterator&gt;\n#include &lt;string_view&gt;\n#include &lt;utility&gt;\n\nstruct split_by_spaces\n{\n    std::string_view      text;\n    static constexpr char delim = ' ';\n\n    struct iterator\n    {\n        const std::string_view&amp; text;\n        std::size_t             cur_pos;\n        std::size_t             end_pos;\n\n        std::string_view operator*() const\n        {\n            return { &amp;text[cur_pos], end_pos - cur_pos };\n        }\n        bool operator==(const iterator&amp; other) const\n        {\n            return cur_pos == other.cur_pos &amp;&amp; end_pos == other.end_pos;\n        }\n        bool operator!=(const iterator&amp; other) const\n        {\n            return !(*this == other);\n        }\n        iterator&amp; operator++()\n        {\n            cur_pos = text.find_first_not_of(delim, end_pos);\n\n            if (cur_pos == std::string_view::npos)\n            {\n                cur_pos = text.size();\n                end_pos = cur_pos;\n                return *this;\n            }\n\n            end_pos = text.find(delim, cur_pos);\n\n            if (cur_pos == std::string_view::npos)\n            {\n                end_pos = text.size();\n            }\n\n            return *this;\n        }\n    };\n\n    [[nodiscard]] iterator begin() const\n    {\n        auto start = text.find_first_not_of(delim);\n        if (start == std::string_view::npos)\n        {\n            return iterator{ text, text.size(), text.size() };\n        }\n        auto end_word = text.find(delim, start);\n        if (end_word == std::string_view::npos)\n        {\n            end_word = text.size();\n        }\n        return iterator{ text, start, end_word };\n    }\n    [[nodiscard]] iterator end() const\n    {\n        return iterator{ text, text.size(), text.size() };\n    }\n};\n\nint main(int argc, char** argv)\n{\n    using namespace std::literals;\n    auto str = &quot; there should be no memory allocation during parsing&quot;\n               &quot;  into words this line and you   should'n create any&quot;\n               &quot;  contaner                  for intermediate words  &quot;sv;\n\n    auto comma = &quot;&quot;;\n    for (std::string_view word : split_by_spaces{ str })\n    {\n        std::cout &lt;&lt; std::exchange(comma, &quot;,&quot;) &lt;&lt; std::quoted(word);\n    }\n\n    auto only_spaces = &quot;                   &quot;sv;\n    for (std::string_view word : split_by_spaces{ only_spaces })\n    {\n        std::cout &lt;&lt; &quot;you will not see this line in output&quot; &lt;&lt; std::endl;\n    }\n}\n</code></pre>\n"}]
['https://Stackoverflow.com/questions/53849', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/1288/']