What is the fastest Pythonic way to remove all stopwords from a list of words in a document? Right now I am using a list comprehension that contains a for loop.from nltk.corpus import stopwords''' Push stopwords to a list '''stop = stopwords.words('english')Document = ' Some huge text .......................... '''' Tokenize the doc '''words = nltk.word_tokenize(Document)''' Comparing two lists '''stopwordsfree_words = [word for word in words if word not in stop]Is there a faster way to do this?
Removing all stopwords from a list of words
python;performance;natural language processing
If stop is a list containing \$s\$ stopwords, and words is a list containing \$w\$ words, then the loop in the list comprehension will be \$O(w s)\$, since it basically has to iterate over both lists in a nested loop.However, if you make the stopwords into a setstop = set(stopwords.words('english')) then each lookup can be done in \$O(1)\$ time.  You would get \$O(w)\$ running time just by changing the data structure like that.Another minor issue is that by convention, Document should be lowercase, because it is a variable rather than a class.