title: Annotate text with BIO-scheme using spaCy
body: <p>I need to annotate a text corpus with BIO-scheme based on a rule method (I have a pre-defined list of tokens and their tags). I'm using spaCy's <code>EntityRuler</code> class for this task. My question is whether there is a neat and efficient way to implement BIO tagging with spaCy? Also, I'm struggling to implement a multi-token BIO tagging:</p>
  
  <pre><code> 'He used sodium-bicarb 5 gr' -&gt; 
  ['O', 'O', 'B-DRUG', 'I-DRUG', 'I-DRUG', 'B-STRENGTH', 'I-STRENGTH')
  </code></pre>
  
  <p>I have a simple (rule-based) script to tag entities I'm interested in:</p>
  
  <pre><code>import numpy as np
  import pandas as pd
  import spacy
  from spacy.pipeline import EntityRuler
  
  nlp = spacy.load('en')
  ruler = EntityRuler(nlp).from_disk('drug_patterns.jsonl')
  nlp.add_pipe(ruler, before='ner')
  
  
  text = 'He has been prescribed ipratropium-albuterol a small dose of 20mg, denzapine and amil-co'
  doc = nlp(text)
  
  for ent in doc.ents:
      print(ent.text, ent.start_char, ent.end_char, ent.label_)
  </code></pre>
  
  <p>The output:</p>
  
  <pre><code>ipratropium 23 34 DRUG
  20mg 61 65 STRENGTH
  denzapine 67 76 DRUG
  amil-co 81 88 DRUG
  </code></pre>
  
  <p>So, I'm not sure how to split <code>'amil-co'</code> into three tags <code>'B-DRUG, I-DRUG and I-DRUG'</code>.</p>
  
  <p>Ideally, I would like to have the following annotation:</p>
  
  <pre><code>    token          BIO
  0   He             O
  1   has            O
  2   been           O
  3   prescribed     O
  4   ipratropium    B-DRUG
  5   -              I-DRUG
  6   albuterol      I-DRUG
  7   a              O
  8   small          O
  9   dose           O
  10  of             O
  11  20             B-STRENGTH
  12  mg             I-STRENGTH
  13  ,              O
  14  denzapine      B-DRUG
  15  and            O
  16  amil           B-DRUG
  17  -              I-DRUG
  18  co             I-DRUG
  19  .              O
  </code></pre>
  
  <p><strong>Also</strong>, in my vocabulary <code>drug_patterns.json</code>, I may have the same long token appear more then once:</p>
  
  <pre><code>{"label": "DRUG", "pattern": [{"lower": "ipratropium"}]}
  {"label": "DRUG", "pattern": [{"lower": "ipratropium"}, {"lower": "bromide"}]}
  {"label": "DRUG", "pattern": [{"lower": "ipratropium"}, {"lower": "-"}, {"lower": "albuterol"}]}
  </code></pre>
  
  <p>which instead of the entire <code>ipratropium-albuterol</code> will pick only the firs (shortest) token <code>ipratropium</code> (as displayed in the output). Is there a simple way to tell spaCy to pick the longest token? </p>
  
  <p>Any ideas will be highly appreciated.</p>