title: explaining NLTK pos_tag ugly mistakes
body: <p>I am doing text mining with Python3-NLTK. in Preprocessing step I wanted to implement Noun-phrase chunking, which requires <a href="http://www.nltk.org/book/ch07.html" rel="nofollow noreferrer">POS-tagging and selection according to a regexp (grammar)</a>. My results were not satisfying, for example I had the phrase <code>the president of</code> because pos_tag had identified it as <code>DT,NN,NNP</code> (how on earth <code>of</code> might be a proper noun!)</p>
  
  <p>so I delved into NLTK pos_tag, and find this peculiar result:</p>
  
  <p>With</p>
  
  <pre><code>import nltk
  nltk.help.upenn_tagset()
  </code></pre>
  
  <p>you'll get a complete list of available tags in NLTK pos_tag. this is one of them:</p>
  
  <blockquote>
    <p>CD: numeral, cardinal :
        mid-1890 nine-thirty forty-two ...</p>
  </blockquote>
  
  <p>As stated, the word <strong>mid-1980</strong> is a CD. now this is what I've got:</p>
  
  <pre><code>from nltk import pos_tag
  
  t1 = 'it happened in the mid-1890s'
  pos_tag(t1.split())[-1] # gives:('mid-1890s', 'NNS')
  
  t2 = 'it happened in the mid-1890'
  pos_tag(t2.split())[-1] # gives:('mid-1890', 'NN')
  
  t3 = 'mid-1890'
  pos_tag(t3.split())[-1] # gives:('mid-1890', 'NN')
  
  t4 = 'mid-1890s'
  pos_tag(t4.split())[-1] # gives:('mid-1890s', 'NNS')
  </code></pre>
  
  <ol>
  <li><p>Isn't this situation odd?!</p></li>
  <li><p>Is there any (probably supervised) method for POS-tagging improvement? I'm working on over 11'000 docs (up to 500 words for each doc)</p></li>
  </ol>