increase speed & lower memory consumption of text normalization function I'm trying to preprocess a large text document. I've written a text normalization function which takes a disproprtionate amount of time and memory. How can I format the function to lower these two?

The time result below was for this example.

t = "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Curabitur pretium tincidunt lacus. Nulla gravida orci a odio. Nullam varius, turpis et commodo pharetra, est eros bibendum elit, nec luctus magna felis sollicitudin mauris. Integer in mauris eu nibh euismod gravida. Duis ac tellus et risus vulputate vehicula. Donec lobortis risus a elit. Etiam tempor. Ut ullamcorper, ligula eu tempor congue, eros est euismod turpis, id tincidunt sapien risus a quam. Maecenas fermentum consequat mi. Donec fermentum. Pellentesque malesuada nulla a mi. Duis sapien sem, aliquet nec, commodo eget, consequat quis, neque. Aliquam faucibus, elit ut dictum aliquet, felis nisl adipiscing sapien, sed malesuada diam lacus eget erat. Cras mollis scelerisque nunc. Nullam arcu. Aliquam consequat. Curabitur augue lorem, dapibus quis, laoreet et, pretium ac, nisi. Aenean magna nisl, mollis quis, molestie eu, feugiat in, orci. In hac habitasse platea dictumst."


This is the script. 

import time
import string
import nltk
import numpy as np

# text preprocessing module, use boolean flags to customize function
def normalize_text(text, lowercase=True, punctuation=True):

    # Step 1: Tokenize
    output = nltk.word_tokenize(text)

    # Step 2: Convert to lowercase (optional)
    if lowercase:
        output = [word.lower() for word in output]

    # Step 3: Remove punctuation:
    if punctuation:
        output = [str(token).translate(str.maketrans('', '', string.punctuation)) for token in output]
        output = [token for token in output if token != '']

    return(output)


Removing all filters and the corresponding if-statements sped up processing by a mere 0.7%.

def normalize_text2(text):

    # Step 1: Tokenize
    output = nltk.word_tokenize(text)

    # Step 2: Convert to lowercase (optional)
    output = [word.lower() for word in output]

    # Step 3: Remove punctuation:
    output = [str(token).translate(str.maketrans('', '', string.punctuation)) for token in output]
    output = [token for token in output if token != '']

    return(output)


Here is the bagged comparison.

times1 = []

for i in range(1000):
    start = time.time()
    tokens = normalize_text(t)
    end = time.time()
    times1.append(end - start)

time1 = np.mean(times1)
print(time1)    

times2 = []

for i in range(1000):
    start = time.time()
    tokens = normalize_text2(t)
    end = time.time()
    times2.append(end - start)

time2 = np.mean(times2)
print(time2) 

print(time2/time1)


Here are the results:

0.0021646411418914796
0.0021491129398345946
0.9928264312470212


Any advice on how to improve further? For example, how could I reduce the number of different list comprehensions, so that the same sequence of text does not need to crunched anew this many times?
You can save a little bit of time by not re-running str.maketrans for each token, since it's always going to produce the same result:

import nltk
from statistics import mean
import string
import time
from typing import List


def normalize_text3(text: str) -&gt; List[str]:
    output: List[str] = []
    punctuation_filter = str.maketrans('', '', string.punctuation)
    for token in nltk.word_tokenize(text):
        token = token.translate(punctuation_filter)
        if not token:
            continue
        output.append(token.lower())
    return output


tested with:

for func in [normalize_text, normalize_text2, normalize_text3]:
    times = []
    for _ in range(1000):
        start = time.time()
        tokens = normalize_text(t)
        end = time.time()
        times.append(end - start)
    print(f"{func.__name__.rjust(15)}: {mean(times)}")


gets me:

dog runs
 normalize_text: 0.003226396322250366
normalize_text2: 0.0032752704620361327
normalize_text3: 0.0030987038612365725


If you want to lower memory consumption, you might consider having this function return a generator rather than a list...