title: Looking for dictionary words in text file using dictionary in python
body: <p>I read the <a href="https://stackoverflow.com/questions/3788870/how-to-check-if-a-word-is-an-english-word-with-python">how to check dictionary words</a> 
  And I got the idea to check my text file using dictionaries.  I have read the <a href="http://pythonhosted.org/pyenchant/tutorial.html" rel="nofollow noreferrer">pyenchant</a> instructions, and I thought that if I use <code>get_tokenizer</code> to give me back all the dictionary words in the text file.</p>
  
  <p>So here is where I'm stuck:  I want my program to give me all groups of dictionary words in the form of a paragraph. As soon as it encounters any junk characters, considers that a paragraph break, and ignores everything from there till it finds X number of consecutive words.</p>
  
  <p>I want it to read a text file in the sequence of <code>filename_nnn.txt</code>, parse it, and write to <code>parsed_filname_nnn.txt</code>.  I have not got around to doing any file manipulation. </p>
  
  <p>What I have so far:</p>
  
  <pre><code>import enchant
  from enchant.tokenize import get_tokenizer, HTMLChunker
  dictSentCheck = get_tokenizer("en_US")
  sentCheck = raw_input("Check Sentense: ")
  
  def check_dictionary():
      outcome = dictCheck.check(wordCheck) 
      test = [w[0] for w in dictSentCheck(sentCheck)]
  </code></pre>
  
  <p>------ sample text -----</p>
  
  <blockquote>
    <blockquote>
      <p>English cricket cuts ties with Zimbabwe Wednesday, 25 June, 2008 text&amp;lt;void(0);&amp;gt;&amp;lt;void(0);&amp;gt; &amp;lt;void(0);&amp;gt;email &amp;lt;void(0);&amp;gt;print EMAIL THIS ARTICLE your name: your email address: recipient's name: recipient's email address: &amp;lt;;&amp;gt;add another recipient your comment: Send Mail&amp;lt;void(0);&amp;gt; close this form &amp;lt;<a href="http://ad.au.doubleclick.net/jump/sbs.com.au/worldnews;sz=300x250;tile=2;ord=123456789?&amp;gt" rel="nofollow noreferrer">http://ad.au.doubleclick.net/jump/sbs.com.au/worldnews;sz=300x250;tile=2;ord=123456789?&amp;gt</a>; The England and Wales Cricket Board (ECB) announced it was suspending all ties with Zimbabwe and was cancelling Zimbabwe's tour of England next year.</p>
    </blockquote>
  </blockquote>
  
  <p>The script should return:</p>
  
  <blockquote>
    <blockquote>
      <p>English cricket cuts ties with Zimbabwe Wednesday</p>
      
      <p>The England and Wales Cricket Board (ECB) announced it was suspending all ties with Zimbabwe and was cancelling Zimbabwe's tour of England next year</p>
    </blockquote>
  </blockquote>
  
  <p>I accepted abarnert's response.  Below is my final script.  Note it is VERY inefficient, and should be cleaned up some.  Also disclaimer I have not coded since college a LONG time ago.</p>
  
  <pre><code>import enchant
  from enchant.tokenize import get_tokenizer
  import os
  
  def clean_files():
      os.chdir("TARGET_DIRECTORY")
      for files in os.listdir("."):
             #get the numbers out file names 
             file_number = files[files.rfind("_")+1:files.rfind(".")]
  
             #Print status to screen
             print "Working on file: ", files
  
             #Read and process original file
             original_file = open("name_"+file_number+".txt", "r+")
             read_original_file = original_file.read();
  
             #Start the parsing of the files
             token_words = tokenize_words(read_original_file)
             parse_result = ('\n'.join(split_on_angle_brackets(token_words,file_number)))
             original_file.close()
  
             #Commit changes to parsed file
             parsed_file = open("name_"+file_number+"_parse.txt", "wb")
             parsed_file.write(parse_result);
             parsed_file.close()
  
  def tokenize_words(file_words):
      tokenized_sentences = get_tokenizer("en_US")
      word_tokens = tokenized_sentences(file_words)
      token_result = [w[0] for w in word_tokens]
      return token_result
  
  def check_dictionary(dict_word):
      check_word = enchant.Dict("en_US")
      validated_word = check_word.check(dict_word)
      return validated_word
  
  def split_on_angle_brackets(token_words, file_number):
      para = []
      bracket_stack = 0
      ignored_words_per_file = open("name_"+file_number+"_ignored_words.txt", "wb")
      for word in token_words:
          if bracket_stack:
              if word == 'gt':
                  bracket_stack -= 1
              elif word == 'lt':
                  bracket_stack += 1
          else:
              if word == 'lt':
                  if len(para) &gt;= 7:
                      yield ' '.join(para)
                  para = []
                  bracket_stack = 1
              elif word != 'amp':
                  if check_dictionary(word) == True:
                      para.append(word)
                      #print "append ", word
                  else:
                         print "Ignored word: ", word
                         ignored_words_per_file.write(word + " \n")
      if para:
          yield ' '.join(para)
  
      #Close opened files
      ignored_words_per_file.close()
  
  clean_files()
  </code></pre>