String Operation to Split on Punctuation I've a function to split text on punctuation and leave the rest of the string (including whitespace in other items of the list):

import unicodedata

def split_on_punc(text):
    chars = list(text)
    i = 0
    start_new_word = True
    output = []
    while i &lt; len(chars):
        char = chars[i]
        if is_punctuation(char):
            output.append([char])
            start_new_word = True
        else:
            if start_new_word:
                output.append([])
            start_new_word = False
            output[-1].append(char)
        i += 1
    ##print(output)
    return ["".join(x) for x in output]

def is_punctuation(char):
    return True if unicodedata.category(char).startswith("P") else False


E.g.

split_on_punctuation("This, this a sentence. with lotsa' puncts!?@ hahaha looo world")


[out]:

['This',
 ',',
 ' this a sentence',
 '.',
 ' with lotsa',
 "'",
 ' puncts',
 '!',
 '?',
 '@',
 ' hahaha looo world']


It looks like a very complicated way to check through each character and then keep track of whether it's a start of a new word or not.

I'm looking for improvements in speed and simplicity.
Billy Mays here for the regex library!

It's API-compatible with the standard Python re module. It's powered by open source, and it's safe for colored fabrics and carpets!

The regex library offers things like \p{Punctuation}, which is actually a shorthand form of p{Punctuation=Yes} which is really a shortening of p{General_category=Punctuation}.

If you can make a Unicode query, regex supports it. Identifiers, Categories, Blocks, Diacritical Marks - it even does Coptic!

It cleans! It brightens! It eliminates odors! All at the same time!

test_data = "This, this a sentence. with lotsa' puncts!?@ hahaha looo world"
print(f"Test data: '{test_data}'")

import regex

PUNCT_RE = regex.compile(r'(\p{Punctuation})')    

print(PUNCT_RE.split(test_data))


The output looks like:

Test data: 'This, this a sentence. with lotsa' puncts!?@ hahaha looo world'
['This', ',', ' this a sentence', '.', ' with lotsa', "'", ' puncts', '!', '', '?', '', '@', ' hahaha looo world']


regex converts your code from a whopping 21 lines to a 1-line method call-- a 2000% reduction! But you gotta call now!  

Here's how to order:

pip install regex