<p>I have string like this</p>

<pre><code> /c SomeText\MoreText "Some Text\More Text\Lol" SomeText
</code></pre>

<p>I want to tokenize it, however I can't just split on the spaces. I've come up with somewhat ugly parser that works, but I'm wondering if anyone has a more elegant design.</p>

<p>This is in C# btw.</p>

<p><strong>EDIT:</strong> My ugly version, while ugly, is O(N) and may actually be faster than using a RegEx.</p>

<pre><code>private string[] tokenize(string input)
{
    string[] tokens = input.Split(' ');
    List&lt;String&gt; output = new List&lt;String&gt;();

    for (int i = 0; i &lt; tokens.Length; i++)
    {
        if (tokens[i].StartsWith("\""))
        {
            string temp = tokens[i];
            int k = 0;
            for (k = i + 1; k &lt; tokens.Length; k++)
            {
                if (tokens[k].EndsWith("\""))
                {
                    temp += " " + tokens[k];
                    break;
                }
                else
                {
                    temp += " " + tokens[k];
                }
            }
            output.Add(temp);
            i = k + 1;
        }
        else
        {
            output.Add(tokens[i]);
        }
    }

    return output.ToArray();            
}
</code></pre>

[{'answer_id': 54876, 'author': 'Community', 'author_id': -1, 'author_profile': 'https://Stackoverflow.com/users/-1', 'pm_score': 3, 'selected': False, 'text': '<p>The Microsoft.VisualBasic.FileIO namespace (in Microsoft.VisualBasic.dll) has a TextFieldParser you can use to split on space delimeted text.  It handles strings within quotes (i.e., "this is one token" thisistokentwo) well.</p>\n\n<p>Note, just because the DLL says VisualBasic doesn\'t mean you can only use it in a VB project.  Its part of the entire Framework.</p>\n'}, {'answer_id': 54880, 'author': 'Craig', 'author_id': 2894, 'author_profile': 'https://Stackoverflow.com/users/2894', 'pm_score': 0, 'selected': False, 'text': '<p>You also might want to look into regular expressions.  That might help you out.  Here is a sample ripped off from MSDN...</p>\n\n<pre><code>using System;\nusing System.Text.RegularExpressions;\n\npublic class Test\n{\n\n    public static void Main ()\n    {\n\n        // Define a regular expression for repeated words.\n        Regex rx = new Regex(@"\\b(?&lt;word&gt;\\w+)\\s+(\\k&lt;word&gt;)\\b",\n          RegexOptions.Compiled | RegexOptions.IgnoreCase);\n\n        // Define a test string.        \n        string text = "The the quick brown fox  fox jumped over the lazy dog dog.";\n\n        // Find matches.\n        MatchCollection matches = rx.Matches(text);\n\n        // Report the number of matches found.\n        Console.WriteLine("{0} matches found in:\\n   {1}", \n                          matches.Count, \n                          text);\n\n        // Report on each match.\n        foreach (Match match in matches)\n        {\n            GroupCollection groups = match.Groups;\n            Console.WriteLine("\'{0}\' repeated at positions {1} and {2}",  \n                              groups["word"].Value, \n                              groups[0].Index, \n                              groups[1].Index);\n        }\n\n    }\n\n}\n// The example produces the following output to the console:\n//       3 matches found in:\n//          The the quick brown fox  fox jumped over the lazy dog dog.\n//       \'The\' repeated at positions 0 and 4\n//       \'fox\' repeated at positions 20 and 25\n//       \'dog\' repeated at positions 50 and 54\n</code></pre>\n'}, {'answer_id': 54916, 'author': 'harpo', 'author_id': 4525, 'author_profile': 'https://Stackoverflow.com/users/4525', 'pm_score': -1, 'selected': False, 'text': '<p><a href="https://stackoverflow.com/questions/54866/best-way-to-parse-space-seperated-text#54880">Craig</a> is right &#8212; use regular expressions.  <a href="http://msdn.microsoft.com/en-us/library/8yttk7sy.aspx" rel="nofollow noreferrer">Regex.Split</a> may be more concise for your needs.</p>\n'}, {'answer_id': 54928, 'author': 'Todd Myhre', 'author_id': 5626, 'author_profile': 'https://Stackoverflow.com/users/5626', 'pm_score': 5, 'selected': True, 'text': '<p>The computer term for what you\'re doing is <a href="http://en.wikipedia.org/wiki/Lexical_analysis" rel="nofollow noreferrer">lexical analysis</a>; read that for a good summary of this common task.</p>\n\n<p>Based on your example, I\'m guessing that you want whitespace to separate your words, but stuff in quotation marks should be treated as a "word" without the quotes.</p>\n\n<p>The simplest way to do this is to define a word as a regular expression:</p>\n\n<pre><code>([^"^\\s]+)\\s*|"([^"]+)"\\s*\n</code></pre>\n\n<p>This expression states that a "word" is either (1) non-quote, non-whitespace text surrounded by whitespace, or (2) non-quote text surrounded by quotes (followed by some whitespace). Note the use of capturing parentheses to highlight the desired text.</p>\n\n<p>Armed with that regex, your algorithm is simple: search your text for the next "word" as defined by the capturing parentheses, and return it. Repeat that until you run out of "words".</p>\n\n<p>Here\'s the simplest bit of working code I could come up with, in VB.NET. Note that we have to check <em>both</em> groups for data since there are two sets of capturing parentheses.</p>\n\n<pre><code>Dim token As String\nDim r As Regex = New Regex("([^""^\\s]+)\\s*|""([^""]+)""\\s*")\nDim m As Match = r.Match("this is a ""test string""")\n\nWhile m.Success\n    token = m.Groups(1).ToString\n    If token.length = 0 And m.Groups.Count &gt; 1 Then\n        token = m.Groups(2).ToString\n    End If\n    m = m.NextMatch\nEnd While\n</code></pre>\n\n<p>Note 1: <a href="https://stackoverflow.com/users/1228/will">Will\'s</a> answer, above, is the same idea as this one. Hopefully this answer explains the details behind the scene a little better :)</p>\n'}, {'answer_id': 55064, 'author': 'FlySwat', 'author_id': 1965, 'author_profile': 'https://Stackoverflow.com/users/1965', 'pm_score': -1, 'selected': False, 'text': '<blockquote>\n  <p>[^\\t]+\\t|"[^"]+"\\t</p>\n</blockquote>\n\n<p>using the Regex definitely looks like the best bet, however this one just returns the whole string. I\'m trying to tweak it, but not much luck so far.</p>\n\n<pre><code>string[] tokens = System.Text.RegularExpressions.Regex.Split(this.BuildArgs, @"[^\\t]+\\t|""[^""]+""\\t");\n</code></pre>\n'}, {'answer_id': 55201, 'author': 'fryguybob', 'author_id': 4592, 'author_profile': 'https://Stackoverflow.com/users/4592', 'pm_score': 2, 'selected': False, 'text': '<p>There is the state machine approach.</p>\n\n<pre><code>    private enum State\n    {\n        None = 0,\n        InTokin,\n        InQuote\n    }\n\n    private static IEnumerable&lt;string&gt; Tokinize(string input)\n    {\n        input += \' \'; // ensure we end on whitespace\n        State state = State.None;\n        State? next = null; // setting the next state implies that we have found a tokin\n        StringBuilder sb = new StringBuilder();\n        foreach (char c in input)\n        {\n            switch (state)\n            {\n                default:\n                case State.None:\n                    if (char.IsWhiteSpace(c))\n                        continue;\n                    else if (c == \'"\')\n                    {\n                        state = State.InQuote;\n                        continue;\n                    }\n                    else\n                        state = State.InTokin;\n                    break;\n                case State.InTokin:\n                    if (char.IsWhiteSpace(c))\n                        next = State.None;\n                    else if (c == \'"\')\n                        next = State.InQuote;\n                    break;\n                case State.InQuote:\n                    if (c == \'"\')\n                        next = State.None;\n                    break;\n            }\n            if (next.HasValue)\n            {\n                yield return sb.ToString();\n                sb = new StringBuilder();\n                state = next.Value;\n                next = null;\n            }\n            else\n                sb.Append(c);\n        }\n    }\n</code></pre>\n\n<p>It can easily be extended for things like nested quotes and escaping.  Returning as <code>IEnumerable&lt;string&gt;</code> allows your code to only parse as much as you need.  There aren\'t any real downsides to that kind of lazy approach as strings are immutable so you know that <code>input</code> isn\'t going to change before you have parsed the whole thing.</p>\n\n<p>See: <a href="http://en.wikipedia.org/wiki/Automata-Based_Programming" rel="nofollow noreferrer">http://en.wikipedia.org/wiki/Automata-Based_Programming</a></p>\n'}]
['https://Stackoverflow.com/questions/54866', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/1965/']