Constructing a lexicon and frequency array from a long string I have a dataset X_train, which is an array where each entry is an email (a string of characters). There are 11,314 emails, each of which is about 500 characters long. (X_train is a processed version of the training data in the newsgroups dataset.)

Ultimately, my goal is to build from scratch a tf-idf function (knowledge of which is probably not necessary for answering my question). To get there, I have constructed a lexicon which contains each unique word in X_train once and only once. My lexicon has 211441 elements. I also need an array where each entry frequency_train[i] is the number of emails in which a given term lexicon_train[i] appears.

I construct the frequency array as follows:

frequency_train = np.zeros(211441)
for i in range(211441):
    count = 0
    for email in X_train:
        if lexicon_train[i] in email:
           count = count + 1
    frequency_train[i] = count


In the same cell, I am also doing something similar with the testing data X_test. I've been running this in Jupyter notebook, and this process takes a while. A previous and very similar task took about 90 minutes. I suspect that I'm doing this task the slowest possible way. Is there a faster way of doing this? I would also welcome answers that explain why this process should take a long time.
For each word in the lexicon you are searching through each email: (11,314 emails) * (60 words/email) * (211441 word lexicon) = lots of comparisons.

Flip it around.  Use collections.Counter.  Get the unique words in each email (use a set()) and then and update the counter.

from collections import Counter

counts = Counter()

for email in x_train:
    words = set(email.split())   # &lt;= or whatever you use to parse the words
    counts.update(words)


This will give you a dict mapping words in the emails to the number of emails they are in.  (11,314 emails) * (60 words/email) = a lot fewer loops.
This probably also recreated the lexicon (e.g. 
counter.keys() should be the lexicon.

On my computer, it takes 7 seconds to generate 115000 random 60-word emails and collect the counts.