remove special character in a List or String
Regular expressions can be used to create a simple tokenizer and normalizer:
from __future__ import annotations
import re

def tokens(text: str) -&gt; list(str):
    &quot;List all the word tokens in a text.&quot;
    return re.findall('[\w]+', text.lower())

assert tokens(&quot;To be, or not to be, that is the question:&quot;) == ['to', 'be', 'or', 'not', 'to', 'be', 'that', 'is', 'the', 'question']

Otherwise, use an established library like spaCy to generate a list of tokens.