title: nltk regular expression tokenizer
body: <p>I tried to implement a regular expression tokenizer with nltk in python, but the result is this:</p>
  
  <pre><code>&gt;&gt;&gt; import nltk
  &gt;&gt;&gt; text = 'That U.S.A. poster-print costs $12.40...'
  &gt;&gt;&gt; pattern = r'''(?x)    # set flag to allow verbose regexps
  ...     ([A-Z]\.)+        # abbreviations, e.g. U.S.A.
  ...   | \w+(-\w+)*        # words with optional internal hyphens
  ...   | \$?\d+(\.\d+)?%?  # currency and percentages, e.g. $12.40, 82%
  ...   | \.\.\.            # ellipsis
  ...   | [][.,;"'?():-_`]  # these are separate tokens; includes ], [
  ... '''
  &gt;&gt;&gt; nltk.regexp_tokenize(text, pattern)
  [('', '', ''), ('', '', ''), ('', '-print', ''), ('', '', ''), ('', '', '')]
  </code></pre>
  
  <p>But the wanted result is this:</p>
  
  <pre><code>['That', 'U.S.A.', 'poster-print', 'costs', '$12.40', '...']
  </code></pre>
  
  <p>Why? Where is the mistake?</p>