title: Spacy spans for parts of tokens
body: <p><em>EDIT:</em></p>
  
  <p>I update this question due to the recent development of the pipeline for the upcoming spaCy 2.0 release. (<a href="https://alpha.spacy.io/usage/processing-pipelines#custom-components" rel="nofollow noreferrer">described here</a>)</p>
  
  <p>the new <code>_</code> property is designed to carry data for custom pipeline components. My input text contains highlighted parts, for example</p>
  
  <blockquote>
    <p>This is an example text <strong>document this is an example text document.
    This is an example text document this is an example</strong> text document.
    This is an example text document this is an e<strong>xamp</strong>le text document.</p>
  </blockquote>
  
  <p>I would like to use spans to keep track of which parts of this document are highlighted. I chose this example to show that the highlighting may span only parts of a token, but also even more than a sentence. As described im my original question below, internally the span seems to be capable of storing intervals on character-level however e.g. the slicing works on token-index-level.</p>
  
  <p>I imagine the new more user friendly pipeline may help achieving this.</p>
  
  <hr>
  
  <p><em>The original question:</em></p>
  
  <p>As far as I understand the model of spaCy there are documents with tokens and spans. I think I have a pretty decent understanding of tokens since this is a concept that I came over in different nlp libs. However, spans I don't really understand.</p>
  
  <p>Spans are for example used to yield sentences from a document. Also, if you slice a document the return will be of type span. </p>
  
  <p>Internally a span consists, apart from other attributes, of</p>
  
  <pre><code>start
  start_char
  end
  end_char
  </code></pre>
  
  <p>where start and end denote the index of the starting and ending token of the span whereas start_char and end_char denote the starting and ending character of the span.</p>
  
  <p>the start_char and end_char attributes are not exposed by the constructor. </p>
  
  <p>My question is: Are spans supposed to be strictly a grouping of tokens or can they be used for denoting everything from a single character of a token to multiple sentences?</p>
  
  <p>If they are not the according data structure, how can I achieve this?</p>