C++ string tokenizing without streams, with certain conditions My concern is mainly the correctness of the algorithm for tokenizing purposes. The code compiles as it is called as a member function in my program. The code does pass my small amount of unit tests for parsing inputs (61 unit tests, all pass). But, I'm slightly stressed out about this part of code, and also paranoid. I could use a pair of foreign mk.I eyeballs looking at my code. This is part of a bigger project and ideally the tokenizing should be bugfree 

please take a look and give thoughts.
:'(

BACKGROUND &amp; problem parameters
objectives of my tokenizer:


cannot use stringstreams, run out of programmemory
input cppstrings will not have newline or any such limiter at the
end(will be chucked out before inputting into
tokenize_input_refactored() ), IT will be as though the input cppstring was gotten with getline()
separate cppstrings into tokens based on single spacebar (delimiter), or the end of the string in regular way ("M4 255" is tokenized into "M4", "255")
if there's no delimiter, take the entire cppstring as token
backToBack delimiters is immediately invalid input, like " " twoSpacebars in row (or more spacebars in row)
cannot begin or end with spacebar(delimiter), immediately invalid input
too large string ( 31 or more true chars) is illegal input
empty string is illegal input
the "garbage strings" that are pushed into the vector "illegalcommand" are later on used for comparison with known words
for parsing purposes. Just to be sure that command won't be parsed as
valid. (tokens amount as well as tokens themselves are checked against expected values)


Possible solution as C++ member-function:

tokensVec is just an empty vector datamember of Gcodeparser object. It will be cleared empty after each round of tokenizing (or parsing, the same same effect).

bool GcodeParser::tokenize_input_refactored(const std::string &amp; rawInput) {

    /*find delimiter == singleSpacebar*/
    auto res = rawInput.find(delimiter);

    /*find if there was twoOrMoreDelimiters backToBack "  " twospacebars */
    string backToBackDelim;
    backToBackDelim += delimiter;
    backToBackDelim += delimiter;
    auto res2 = rawInput.find(backToBackDelim);

    int searchInd = 0;
    int pos = 0;
    int size = rawInput.length();
    bool returnableResult = false;

    /*if the input was too large =&gt; reject as illegal*/
    if (size &gt; maxCharAmount) {
        tokensVec.push_back("INVALID_COMMAND!");
        return false;
    }
    /*check if empty string*/
    if (size == 0) {
        tokensVec.push_back("INVALID_COMMAND!");
        return false;
    }

    /*found 2delim backtoback*/
    if (res2 != string::npos) {
        tokensVec.push_back("INVALID_COMMAND!");
        return false;
    }

    /*input ended in delimiter or started with delimiter*/
    if (rawInput[0] == delimiter || rawInput[size-1] == delimiter) {
        tokensVec.push_back("INVALID_COMMAND!");
        return false;
    }

    /*delimiter wasnt found =&gt; take complete word into vec for later parsing*/
    if (res == string::npos) {
        tokensVec.push_back(rawInput);
        return true;
    }
    else {
        while (pos != string::npos) {
            pos = rawInput.find(delimiter, searchInd);
            if (pos != string::npos) {
                tokensVec.push_back(rawInput.substr(searchInd, pos - searchInd));
                searchInd = pos + 1;
            }
        }
        /*take the last valid token*/
        tokensVec.push_back( rawInput.substr(searchInd, size - searchInd) );
        return true;
    }
}
Well, it seems to do what you want. But, we can do better.

Review:


Avoid dynamic allocation where possible.

string backToBackDelim;
backToBackDelim += delimiter;
backToBackDelim += delimiter;
auto res2 = rawInput.find(backToBackDelim)


becomes

const char backToBackDelim[] = {delimiter, delimiter};
auto res2 = rawInput.find(backToBackDelim, 0, 2)

Don't pass std::string by constant reference unless you depend on the terminator. Pass a view by value instead, specifically C++17 std::string_view. It is more flexible and tends to be more efficient.
An empty line directly after the prototype is curious. Either remove it, or move the opening brace { for the function there.
Define your variables in the smallest scope you can, and invest in good names. What is res / res2, and why do you initialize it all the way up there, and only use it the once down here?
You know what, eliminate it completely, it's bloat.
Choosing good names should also be applied to the function itself. Yours seems to be a placeholder.
Consider marking things const to ease comprehension. No Need to search for possible sites of modification then. That is, if having that extra-variable actually pays for itself with clarity.
You have a bunch of conditions under which you error out. The curious Thing is that you manually duplicated the handling block. Consider merging the conditions so you only need it once.
Don't use else if control-flow doesn't resume after the code-block anyway. Less indentation is better.
You know .substr() will adjust the second argument, if provided, to not leave the string? So, don't try so hard in your last call.


Re-Design:

Mull over your spec. And now try to simplify it. Doing so leads to a simpler, more efficient design, which is optimized for legal input:


Too large or empty => illegal.
Tokenize.
Empty token => illegal.


Rewrite:

bool GcodeParser::tokenize_input_refactored(std::string_view s) {
    // Consider adding here: tokensVec.clear();
    if (auto pos = size(s); pos &amp;&amp; pos &lt;= maxCharAmount) {
        while ((pos = s.find(delimiter))) {
            tokensVec.push_back(s.substr(0, pos));
            if (pos == std::string_view::npos)
                return true;
            s = s.substr(pos + 1);
        }
        tokensVec.clear();
    }
    tokensVec.push_back("INVALID_COMMAND!");
    return false;
}