title: How to handle a tokenize error with unterminated multiline comments (python 2.6)
body: <p>The following sample code:</p>
  
  <pre><code>import token, tokenize, StringIO
  
  def generate_tokens(src):
      rawstr = StringIO.StringIO(unicode(src))
      tokens = tokenize.generate_tokens(rawstr.readline)
      for i, item in enumerate(tokens):
          toktype, toktext, (srow,scol), (erow,ecol), line = item
          print i, token.tok_name[toktype], toktext
  
  s = \
  """
   def test(x):
       \"\"\" test with an unterminated docstring
  """
  
  generate_tokens(s)
  </code></pre>
  
  <p>causes the following to fire:</p>
  
  <pre><code>... (stripped a little)
  File "/usr/lib/python2.6/tokenize.py", line 296, in generate_tokens
      raise TokenError, ("EOF in multi-line string", strstart)
  tokenize.TokenError: ('EOF in multi-line string', (3, 5))
  </code></pre>
  
  <p><strong>Some questions about this behaviour:</strong></p>
  
  <ol>
  <li>Should I catch and 'selectively' ignore tokenize.TokenError here? Or
  should I stop trying to generate tokens from non-compliant/non-complete code? If so, how would I check for that?</li>
  <li>Can this error (or similar errors) be caused by anything other than an
  unterminated docstring?</li>
  </ol>