Filtering textual data in Python
<p>I'm having a problem understanding what I'm doing wrong here. I have the code below (fairly straightforward). </p>

<pre><code>def compileWordList(textList, wordDict):
    '''Function to extract words from text lines exc. stops,
        and add associated line nums'''
    i = 0;
    for row in textList:
        i = i + 1
        words = re.split('\W+', row)
        for wordPart in words:
            word = repr(wordPart)
            word = word.lower()
            if not any(word in s for s in stopsList):
                if word not in wordDict:
                    x = wordLineNumsContainer()
                    x.addLineNum(i)
                    wordDict[word] = x
                elif word in wordDict:
                    lineNumValues = wordDict[word]
                    lineNumValues.addLineNum(i)
                    wordDict[word] = lineNumValues
            elif any(word in s for s in stopsList):
                print(word)
</code></pre>

<p>The code takes a string (sentence) from a list. It then splits the string for whole words using a re.split() method, returning a list of strings (the words).</p>

<p>I then force the string to lower case. Then I want it to check the word for existence in a list of stop-words I have (words too common in English to bother with). The part that checks if the <code>word</code> is in <code>stopsList</code> never seems to work, as stops words end up in my <code>wordDict</code> every time. Also I added the bottom <code>print(word)</code> statement in order to check it was catching them, but nothing ever gets printed :(</p>

<p>There are hundreds of stop words being used in the strings passing through.</p>

<p>Please can someone enlighten me here? Why are the strings never getting filtered for being stop words?</p>

<p>Many thanks,
Alex</p>
