title: Discovering Poetic Form with NLTK and CMU Dict
body: <p><strong>Edit: This code has been worked on and released as a basic module: <a href="https://github.com/hyperreality/Poetry-Tools" rel="noreferrer">https://github.com/hyperreality/Poetry-Tools</a></strong></p>
  
  <p>I'm a linguist who has recently picked up python and I'm working on a project which hopes to automatically analyze poems, including detecting the form of the poem.  I.e. if it found a 10 syllable line with 0101010101 stress pattern, it would declare that it's iambic pentameter.  A poem with 5-7-5 syllable pattern would be a haiku.</p>
  
  <p>I'm using the following code, part of a larger script, but I have a number of problems which are listed below the program:</p>
  
  <p>corpus in the script is simply the raw text input of the poem.</p>
  
  <pre><code>import sys, getopt, nltk, re, string
  from nltk.tokenize import RegexpTokenizer
  from nltk.util import bigrams, trigrams
  from nltk.corpus import cmudict
  from curses.ascii import isdigit
  
  ...
  
  def cmuform():
      tokens = [word for sent in nltk.sent_tokenize(corpus) for word in nltk.word_tokenize(sent)]
      d = cmudict.dict()
      text = nltk.Text(tokens)
      words = [w.lower() for w in text]
      regexp = "[A-Za-z]+"
      exp = re.compile(regexp)
  
      def nsyl(word):
          lowercase = word.lower()
          if lowercase not in d:
                  return 0
          else:
              first = [' '.join([str(c) for c in lst]) for lst in max(d[lowercase])]
              second = ''.join(first)
              third = ''.join([i for i in second if i.isdigit()]).replace('2', '1')
              return third 
                  #return max([len([y for y in x if isdigit(y[-1])]) for x in d[lowercase]])      
  
      sum1 = 0
      for a in words:
              if exp.match(a):
              print a,nsyl(a),
                  sum1 = sum1 + len(str(nsyl(a)))
  
      print "\nTotal syllables:",sum1
  </code></pre>
  
  <p>I guess that the output that I want would be like this:</p>
  
  <p>1101111101</p>
  
  <p>0101111001</p>
  
  <p>1101010111</p>
  
  <p>The first problem is that I lost the line breaks during the tokenization, and I really need the line breaks to be able to identify form.  This should not be too hard to deal with though.  The bigger problems are that:</p>
  
  <ul>
  <li>I can't deal with non-dictionary words. At the moment I return 0 for them, but this will confound any attempt to identify the poem, as the syllabic count of the line will probably decrease.</li>
  <li>In addition, the CMU dictionary often says that there is stress on a word - '1' - when there is not - '0 - . Which is why the output looks like this: 1101111101, when it should be the stress of iambic pentameter: 0101010101<br>
  So how would I add some fudging factor so the poem still gets identified as iambic pentameter when it only approximates the pattern?  It's no good to code a function that identifies lines of 01's when the CMU dictionary is not going to output such a clean result.  I suppose I'm asking how to code a 'partial match' algorithm.</li>
  </ul>