Chunk a large set of characters by a specified set of delimiters and a maximum chunk size Problem
I need to be able to grab a subset of characters but ensure that each set doesn't exceed a specified limit and must end with a defined set of characters. For example, say I have an entire book worth of characters, I need to chunk that into a set of 300 characters but ensure that none of those chunks is split mid-sentence.
To put it into pesudo code I basically need:
foreach(var chunk in text.Chunk(x).ButAlsoSplitOnFirst('.', '!', '?', etc...);

My solution
public static class StringExtensions
    {
        public static IEnumerable&lt;string&gt; ChunkWithDelimeters(this string source, int maxChar, params char[] delimeters)
        {
            var sourceSpan = source.AsSpan();
            var items = new List&lt;string&gt;();

            var pointer = 0;
            var lazyPointer = 0;

            while (pointer &lt;= sourceSpan.Length)
            {
                bool foundMatch = false;

                pointer = Math.Min(lazyPointer + maxChar, sourceSpan.Length);

                if (pointer == sourceSpan.Length)
                {
                    items.Add(sourceSpan.Slice(lazyPointer, pointer - lazyPointer).TrimStart().ToString());
                    break;
                }

                for (var j = pointer; j &gt;= lazyPointer; j--)
                {
                    var tempChar = sourceSpan[j];

                    if (delimeters.Contains(tempChar))
                    {
                        foundMatch = true;
                        items.Add(sourceSpan.Slice(lazyPointer, j + 1 - lazyPointer).TrimStart().ToString());
                        lazyPointer = j + 1;
                    }
                }

                if (!foundMatch)
                {
                    items.Add(sourceSpan.Slice(lazyPointer, pointer - lazyPointer).Trim().ToString());
                    lazyPointer = pointer;
                }
            }

            return items;
        }
    }

Benchmarks




source.Length
maxChar
delimiters
Mean (μs)
Error (μs)
StdDev (μs)




7320000
30
'.'
73,622.739
1,412.4035
1,252.0589


439200
30
'.'
4,122.574
68.6392
60.8468


731
30
'.'
5.125
0.0466
0.0436



My question
What is the optimal solution to this problem? Is there any way I can improve the performance of my solution?
Don't you need to test for delimiters here?  From reading your code I assume we want to max chunk but if inside the chunk there are delimiters we want to break that up.  If the last section fits in the entire chunk it's not checking for delimiters
if (pointer == sourceSpan.Length)
{
    items.Add(sourceSpan.Slice(lazyPointer, pointer - lazyPointer).TrimStart().ToString());
    break;
}

Also you could use IndexOfAny instead of the loop at the end
WARNING this code not tested and just used an example
var slice = sourceSpan.Slice(lazyPointer, pointer - lazyPointer);
var contains = slice.IndexOfAny(delimeters);
while (contains &gt; -1)
{
    items.Add(slice.Slice(0, contains + 1).ToString());
    slice = slice.Slice(contains + 1);
    contains = slice.IndexOfAny(delimeters);
}

Now some other points that I think you should split by delimiters first then chunks.  Instead of chunks then delimiters.  If I have a string of  &quot;12345.6789.8.654321&quot; and split by 7.   The first two coming out would be &quot;12345&quot; &lt;-- looks good but then the next chunk would be &quot;6&quot; then chunk after that would be &quot;789&quot;,&quot;8&quot;,&quot;6&quot;,&quot;54321&quot;  if we split by delimiters first then chunked them it would output what I think most people would expect.