title: Use NLTK RegexTokenizer to tokenize text and write it to a CSV
body: <p>Thank you in advance for your help. I'm extremely lost. I'm trying to import a corpus and then have it print trigrams into a csv file with the frequency distribution and the relative frequency in two columns next to the first column that contains the entire trigram. But I don't understand RegexTokenizer well enough to do it. The code below gets 90% of the way there but the RegexTokenizer only looks for letters so it splits phrases with conjunctions like "don't go away" into trigrams: "don t go" </p>
  
  <p>I need it to stop doing that. Without the RegexTokenizer the trigrams look like this: (u'middle', u'class', u'americans') I'd imagine you can use RegexTokenizer to find only the phrases between u' and ' but I don't know how to do that. </p>
  
  <pre><code>import nltk
  import re
  from nltk.corpus.reader.plaintext import PlaintextCorpusReader
  from nltk import FreqDist
  import math 
  from decimal import *
  from nltk.tokenize import RegexpTokenizer, WhitespaceTokenizer
  import csv
  
  #this imports the text files in the folder into corpus called speeches
  corpus_root = '/Users/root...'
  speeches = PlaintextCorpusReader(corpus_root, '.*\.txt')
  
  print "Finished importing corpus"
  
  tokenizer = RegexpTokenizer(r'\w+')
  raw = speeches.raw().lower()
  tokens = tokenizer.tokenize(raw)
  tgs = nltk.trigrams(tokens)
  fdist = nltk.FreqDist(tgs)
  minscore = 200
  numwords = len(raw)
  c = csv.writer(open("TPNngrams.csv", "wb"))
  for k,v in fdist.items():
      if v &gt; minscore:
          rf = Decimal(v)/Decimal(numwords)
          firstword, secondword, thirdword = k
          trigram = firstword + " " + secondword + " " + thirdword
          results = trigram,v,rf
          c.writerow(results)
          print firstword, secondword, thirdword, v, rf
  </code></pre>
  
  <p>And I also keep getting this error randomly: </p>
  
  <pre><code>UnicodeEncodeError: 'ascii' codec can't encode character u'\xa9' in position 0: ordinal not in range(128)
  </code></pre>