prompt: We are currently solving the following issue within our repository. Here is the issue text:
  --- BEGIN ISSUE ---
  Bug in tokenizer?
  Here's a minimum viable code to reproduce:
  
  ```
  from flair.data import Sentence
  from flair.models import SequenceTagger
  
  model = SequenceTagger.load("ner-ontonotes-fast")
  full_text = "\"In the 1960s and 1970s...\" Then came Thierry Mugler and Gianni Versace."
  sentence = Sentence(full_text, use_tokenizer=True)
  model.predict(sentence)
  print(f"full text  : {full_text}")
  print(f"text length: {len(full_text)}")
  print("tag\tstart\tend\tto_original_text()")
  for entity in sentence.get_spans('ner'):
      print(f"{entity.tag}\t{entity.start_pos}\t{entity.end_pos}\t{entity.to_original_text()}")
  ```
  
  Output:
  
  ``` $  python predict.py 
  full text  : "In the 1960s and 1970s..." Then came Thierry Mugler and Gianni Versace.
  text length: 72
  tag	start	end	to_original_text()
  DATE	8	13	1960s
  DATE	18	23	1970s
  PERSON	81	94	ThierryMugler
  PERSON	97	110	GianniVersace
  ```
  Seems the resulting tokens have start_pos and end_pos indexes larger than the real text length. Note also that the method to_original_text() is eating the spaces, so I suppose it is related.
  
  Any ideas about what is causing the trouble?
  
  --- END ISSUE ---
  
  Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
  --- BEGIN FILES ---
  Path: `setup.py`
  Content:
  ```
  1 from setuptools import setup, find_packages
  2 
  3 setup(
  4     name='flair',
  5     version='0.3.2',
  6     description='A very simple framework for state-of-the-art NLP',
  7     long_description=open("README.md", encoding='utf-8').read(),
  8     long_description_content_type="text/markdown",
  9     author='Alan Akbik',
  10     author_email='alan.akbik@zalando.de',
  11     url='https://github.com/zalandoresearch/flair',
  12     packages=find_packages(exclude='test'),  # same as name
  13     license='MIT',
  14     install_requires=[
  15         'torch==0.4.1',
  16         'gensim==3.4.0',
  17         'typing==3.6.4',
  18         'tqdm==4.23.4',
  19         'segtok==1.5.6',
  20         'matplotlib==3.0.0',
  21         'mpld3==0.3',
  22         'sklearn',
  23         'sqlitedict==1.6.0',
  24         'deprecated==1.2.4',
  25     ],
  26     include_package_data=True,
  27     python_requires='>=3.6',
  28 )
  29
  ```
  --- END FILES ---
  
  Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.
  
  Here is an example:
  
  ```diff
  diff --git a/examples/server_async.py b/examples/server_async.py
  --- a/examples/server_async.py
  +++ b/examples/server_async.py
  @@ -313,4 +313,4 @@
   
   
   if __name__ == "__main__":
  -    asyncio.run(run_async_server("."), debug=True)
  +    asyncio.run(run_async_server(), debug=True)
  diff --git a/examples/server_sync.py b/examples/server_sync.py
  --- a/examples/server_sync.py
  +++ b/examples/server_sync.py
  @@ -313,5 +313,5 @@
   
   
   if __name__ == "__main__":
  -    server = run_sync_server(".")
  +    server = run_sync_server()
       server.shutdown()
  
  ```
  
golden_diff: diff --git a/setup.py b/setup.py
  --- a/setup.py
  +++ b/setup.py
  @@ -15,8 +15,8 @@
           'torch==0.4.1',
           'gensim==3.4.0',
           'typing==3.6.4',
  -        'tqdm==4.23.4',
  -        'segtok==1.5.6',
  +        'tqdm==4.26.0',
  +        'segtok==1.5.7',
           'matplotlib==3.0.0',
           'mpld3==0.3',
           'sklearn',
  
verification_info: {"golden_diff": "diff --git a/setup.py b/setup.py\n--- a/setup.py\n+++ b/setup.py\n@@ -15,8 +15,8 @@\n         'torch==0.4.1',\n         'gensim==3.4.0',\n         'typing==3.6.4',\n-        'tqdm==4.23.4',\n-        'segtok==1.5.6',\n+        'tqdm==4.26.0',\n+        'segtok==1.5.7',\n         'matplotlib==3.0.0',\n         'mpld3==0.3',\n         'sklearn',\n", "issue": "Bug in tokenizer?\nHere's a minimum viable code to reproduce:\r\n\r\n```\r\nfrom flair.data import Sentence\r\nfrom flair.models import SequenceTagger\r\n\r\nmodel = SequenceTagger.load(\"ner-ontonotes-fast\")\r\nfull_text = \"\\\"In the 1960s and 1970s...\\\" Then came Thierry Mugler and Gianni Versace.\"\r\nsentence = Sentence(full_text, use_tokenizer=True)\r\nmodel.predict(sentence)\r\nprint(f\"full text  : {full_text}\")\r\nprint(f\"text length: {len(full_text)}\")\r\nprint(\"tag\\tstart\\tend\\tto_original_text()\")\r\nfor entity in sentence.get_spans('ner'):\r\n    print(f\"{entity.tag}\\t{entity.start_pos}\\t{entity.end_pos}\\t{entity.to_original_text()}\")\r\n```\r\n\r\nOutput:\r\n\r\n``` $  python predict.py \r\nfull text  : \"In the 1960s and 1970s...\" Then came Thierry Mugler and Gianni Versace.\r\ntext length: 72\r\ntag\tstart\tend\tto_original_text()\r\nDATE\t8\t13\t1960s\r\nDATE\t18\t23\t1970s\r\nPERSON\t81\t94\tThierryMugler\r\nPERSON\t97\t110\tGianniVersace\r\n```\r\nSeems the resulting tokens have start_pos and end_pos indexes larger than the real text length. Note also that the method to_original_text() is eating the spaces, so I suppose it is related.\r\n\r\nAny ideas about what is causing the trouble?\n", "before_files": [{"content": "from setuptools import setup, find_packages\n\nsetup(\n    name='flair',\n    version='0.3.2',\n    description='A very simple framework for state-of-the-art NLP',\n    long_description=open(\"README.md\", encoding='utf-8').read(),\n    long_description_content_type=\"text/markdown\",\n    author='Alan Akbik',\n    author_email='alan.akbik@zalando.de',\n    url='https://github.com/zalandoresearch/flair',\n    packages=find_packages(exclude='test'),  # same as name\n    license='MIT',\n    install_requires=[\n        'torch==0.4.1',\n        'gensim==3.4.0',\n        'typing==3.6.4',\n        'tqdm==4.23.4',\n        'segtok==1.5.6',\n        'matplotlib==3.0.0',\n        'mpld3==0.3',\n        'sklearn',\n        'sqlitedict==1.6.0',\n        'deprecated==1.2.4',\n    ],\n    include_package_data=True,\n    python_requires='>=3.6',\n)\n", "path": "setup.py"}], "after_files": [{"content": "from setuptools import setup, find_packages\n\nsetup(\n    name='flair',\n    version='0.3.2',\n    description='A very simple framework for state-of-the-art NLP',\n    long_description=open(\"README.md\", encoding='utf-8').read(),\n    long_description_content_type=\"text/markdown\",\n    author='Alan Akbik',\n    author_email='alan.akbik@zalando.de',\n    url='https://github.com/zalandoresearch/flair',\n    packages=find_packages(exclude='test'),  # same as name\n    license='MIT',\n    install_requires=[\n        'torch==0.4.1',\n        'gensim==3.4.0',\n        'typing==3.6.4',\n        'tqdm==4.26.0',\n        'segtok==1.5.7',\n        'matplotlib==3.0.0',\n        'mpld3==0.3',\n        'sklearn',\n        'sqlitedict==1.6.0',\n        'deprecated==1.2.4',\n    ],\n    include_package_data=True,\n    python_requires='>=3.6',\n)\n", "path": "setup.py"}]}