Remove words from an exclude list in each row of a pandas.Series
You currently have to loop over all exclude words to check if it is equal to the current word. If your exclude was a set, you could just do if j in exclude. This change alone should speed up your code a lot (how much depends on how large your list of exclude words actually is).

In addition, you could simplify getting the exclude words:

from collections import Counter
form itertools import takewhile

THRESHOLD = 25

words = Counter(whatever_generates_your_words())
exclude = set(t[0] for t in takewhile(lambda t: t[1] &lt;= THRESHOLD,
                                      reversed(words.most_common())))


This uses the fact that collections.Counter.most_common is sorted by frequency, so reversing it starts with the least common words. itertools.takewhile stops taking when the condition is no longer true, so you don't need to go through all words in the Counter.

For the filtering you should probably use some list comprehensions. All three of the following functions are doing the same thing:

from itertools import filterfalse

def remove_nonfrequent(note):
    sentences = []
    for sentence in note.split('. '):
        sentence_ = []
        for word in sentence:
            if word not in exclude:
                sentence_.append(word)
        if sentence:
            sentences.append(" ".join(sentence))
    return ". ".join(sentences)

def remove_nonfrequent(note):
    return ". ".join(" ".join(filterfalse(exclude.__contains__,
                              sentence.split(" ")))
                     for sentence in note.split('. '))
                     if sentence)

def remove_nonfrequent(note):        
    return ". ".join(" ".join(word for word in sentence.split(" ")
                              if word not in exclude)
                     for sentence in note.split('. ') if sentence)


I personally prefer the last one, as it is the most readable one to me.

Note that Python has an official style-guide, PEP8, which recommends using 4 spaces as indentation and surrounding operators with spaces.



Since you are using this function with pandas later, a different approach might actually be faster. There is a pd.Series.replace method that can take a dictionary of replacements. You can use it like this:

from itertools import repeat

replacements = dict(zip((fr'\b{word}\b' for word in exclude), repeat("")))
df.NOTES.replace(replacements, regex=True, inplace=True)
df.NOTES.replace({r' +': ' ', r' +\.': '.'}, regex=True, inplace=True)


The first replace does all the word replacements (using \b to ensure only complete words are replaced), while the latter one fixes multiple spaces and spaces before a period.