How to remove special characters from txt files using Python
<pre><code>from glob import glob
pattern = "D:\\report\\shakeall\\*.txt"
filelist = glob(pattern)
def countwords(fp):
    with open(fp) as fh:
        return len(fh.read().split())
print "There are" ,sum(map(countwords, filelist)), "words in the files. " "From directory",pattern
import os
uniquewords = set([])
for root, dirs, files in os.walk("D:\\report\\shakeall"):
    for name in files:
        [uniquewords.add(x) for x in open(os.path.join(root,name)).read().split()]
print "There are" ,len(uniquewords), "unique words in the files." "From directory", pattern
</code></pre>

<p>So far my code is this. This counts the number of unique words and total words from <code>D:\report\shakeall\*.txt</code></p>

<p>The problem is, for example, this code recognizes <code>code</code> <code>code.</code> and <code>code!</code> different words. So, this can't be an answer to an exact number of unique words.</p>

<p>I'd like to remove special characters from 42 text files using Windows text editor</p>

<p>Or make an exception rule that solve this problem.</p>

<p>If using the latter, how shoud I make up my code?</p>

<p>Make it to directly modify text files? Or make an exception that doesn't count special characters?</p>
