      h4 Adding a new class to a pre-trained model
  
      p spaCy provides a named entity recogniser trained on OntoNotes 5, that recognises the usual classes --- person, location, organization, etc. In this tutorial we'll look for video games, using comments posted to Reddit in 2015. The Reddit comment corpus is some of my favourite data. It's a good amount of text, and it's quite clean. We don't have to do any HTML or MediaWiki processing, and there's very little automatically generated text. Most of the comments are around the same length, and people mostly type in full sentences.
  
      p First, let's have a look at what spaCy does out-of-the-box, with the pre-trained statistical model. Install spaCy and download the data:
  
      pre
          code
              | $ pip install spacy && python -m spacy.en.download all
  
 
</s> remove     p Until machines can think for themselves, we have to either program them explicitly, or give them examples.  Let's say you want to write a program that tags mentions of a certain type of thing in text --- let's say, food.  You can't write this program without supplying some definition of what you consider to be food, and what you don't. That much is fundamental. But, you don't necessarily need to sit down with an arbitrary sample of text, and start tagging it manually, word-by-word. Sometimes you do --- but sometimes there are easier ways.
    
    p In this post I'll describe how to adapt spaCy's named entity recogniser to recognise categories of your choosing, including a quick and dirty way to produce training data semi-automatically. The strategy is to define a handful of seed words or phrases, and then use word2vec to generate a more expansive list. Before using Gensim to train the word vectors, we merge base noun phrases and previously recognised entities into single tokens, using spaCy's annotations. This gives a vector space that includes phrases as well as words. To train our tagger, we can manually classify some of these phrases in context, or we can use an old data generation trick adapted from word sense disambiguation.

 </s> add  </s> remove     p spaCy provides a named entity recogniser trained on OntoNotes 5, that recognises the usual classes --- person, location, organization, etc. In this tutorial we'll look for video games, using comments posted to Reddit in 2015. The Reddit comment corpus is some of my favourite data. It's a good amount of text, and it's quite clean. We don't have to do any HTML or MediaWiki processing, and there's very little automatically generated text. Most of the comments are around the same length, and people mostly type in full sentences.
 </s> add     p spaCy provides a named entity recogniser trained on OntoNotes 5, that recognises the usual classes --- person, location, organization, etc. In this tutorial I'll show how to:

    ol
        li Add classes to an existing entity recognition model;
        li Update the model to fix errors on your data;
        li Train a new named entity recognition model from scratch;

    h4 How the entity recognizer works

    p Named entity recognition is typically modelled as a problem of assigning labels to contiguous spans of text. The spans are usually encoded into structured tags that indicate the span boundaries and types, so that the entities can be assigned like part-of-speech tags, using a linear-chain CRF or a Hidden Markov Model.

    p All three parts of this design are bad. Named entities are not actually flat-structured. For instance, #[the University of Sydney] is an organization name that contains a city name. This is very common. Entities also don't fall into neat exclusive hierarchies. Finally, and of least significance, encoding the problem into per-word tags fails to make the right thing easy. It makes it tricky to write loss functions and features that refer to the actual structures we're trying to predict: the spans.

    p Addressing the first two problems would require us to depart significantly from most of the existing data and research.  We'll have richly labelled, internally structured entities eventually. But we can at least set up the learning problem better, without the pointless tag encoding.

    p Instead of a tagger, spaCy's entity recognizer is an incremental transition-based parser, with a simple transition system that matches the #[code BILOU] tagging scheme. We have the following actions: </s> remove     p It's important that we're able to do this over #[em phrases], not just individual words. We want to be tagging instances of "hot dog", not just "dog". A simple way to achieve this is to use spaCy's syntactic parser to identify base noun phrases. We then merge these noun phrases into single tokens, before we send the data to Gensim's word2vec implementation. I did this over all comments posted to Reddit in 2015.
 </s> add     p The model has learned that as a noun, #[em take] is similar to phrases like #[em personal stance] and #[em best guess]. Its similarity to the verbal sense is quite low:
    ul
        li Begin(T): Open a new entity of type T at position i. Advance to position i+1. Only valid if no entities are currently open.
        li In(T): Advance to position i+1. Only valid if an entity of type T is currently open.
        li Last(T): Close the current open entity, setting its end-point to i+1. Advance to i+1. Only valid if an entity of type T is currently open.
        li Unitary(T): Add an entity of type T, beginning at position i and ending at i+1. Advance to position i+1. Only valid if no entities are currently open.
        li Out: Advance to i+1. Only valid if an entity is not currently open.

    p The BILOU system introduces some redundancy, to make the decision boundary between the classes a little sharper. The idea is that this might make them a bit more linearly separable. If an entity of type T is currently open, the relevant decision is In(T) vs Last(T). If no entity is open, the decision is Begin(x) vs. Unitary(x) vs Out. The accuracy advantage of this over the IOB scheme is well observed.

    p A well set up tagging model will be equivalent to our parsing model. It's not very difficult to write your tagger such that invalid tag sequences are given zero probability. But to do that, you usually have to break the abstraction provided by the tagging set up. The linear chain CRF or HMM model invites you to consider the tags you're assigning as atomic. They're not --- they're structured. Similarly, unless you break abstraction, you'll probably write suboptimal features. The standard approach invites an Nth-order Markov assumption: you only get to ask questions about the last N tags. A typical N is 2 or 3. This means you can't ask a simple question, like "What's the first word of the current entity?". This is a natural question to ask, and a model that stops you from asking it is nonsense.
    
   p So: spaCy's entity recognizer is an instance of class #[code spacy.parser.Parser]. It holds an instance of #[code spacy.parser.ner.BiluoPushDown], which owns an array of #[code Transition] structs. Each struct holds three function pointers, that are used to apply the action, determine whether the action is valid given the current state, and determine what its cost would be with respect to a gold-standard analysis.
   
   p Entity recognition proceeds word-by-word. At each word we ask the model to score all the actions. We then loop over the structs and ask each to tell us whether they are valid. We then apply the highest scoring valid action, updating the state in place. At the end of the sentence we apply the analysis to the document.
   
   p We can condition the decisions on the whole document, and on all previous decisions. So we can easily ask questions about what entities have been frequent, and how we previously tagged some word. We don't currently have any features that do this, although research suggests they can be effective. We can also easily ask any questions about the current open entity, including whether it currently matches a gazetteer.

   p The parser's model employs sparse data structures, so we don't have to resize anything if we add an additional class, and start providing examples of it. This is probably not an optimal way to fit the model, but it's often convenient.
      h4 Adding a new class to a pre-trained model
  
      p spaCy provides a named entity recogniser trained on OntoNotes 5, that recognises the usual classes --- person, location, organization, etc. In this tutorial we'll look for video games, using comments posted to Reddit in 2015. The Reddit comment corpus is some of my favourite data. It's a good amount of text, and it's quite clean. We don't have to do any HTML or MediaWiki processing, and there's very little automatically generated text. Most of the comments are around the same length, and people mostly type in full sentences.
  
    ul
        li Begin(T): Open a new entity of type T at position i. Advance to position i+1. Only valid if no entities are currently open.
        li In(T): Advance to position i+1. Only valid if an entity of type T is currently open.
        li Last(T): Close the current open entity, setting its end-point to i+1. Advance to i+1. Only valid if an entity of type T is currently open.
        li Unitary(T): Add an entity of type T, beginning at position i and ending at i+1. Advance to position i+1. Only valid if no entities are currently open.
        li Out: Advance to i+1. Only valid if an entity is not currently open.

    p The BILOU system introduces some redundancy, to make the decision boundary between the classes a little sharper. The idea is that this might make them a bit more linearly separable. If an entity of type T is currently open, the relevant decision is In(T) vs Last(T). If no entity is open, the decision is Begin(x) vs. Unitary(x) vs Out. The accuracy advantage of this over the IOB scheme is well observed.

    p A well set up tagging model will be equivalent to our parsing model. It's not very difficult to write your tagger such that invalid tag sequences are given zero probability. But to do that, you usually have to break the abstraction provided by the tagging set up. The linear chain CRF or HMM model invites you to consider the tags you're assigning as atomic. They're not --- they're structured. Similarly, unless you break abstraction, you'll probably write suboptimal features. The standard approach invites an Nth-order Markov assumption: you only get to ask questions about the last N tags. A typical N is 2 or 3. This means you can't ask a simple question, like "What's the first word of the current entity?". This is a natural question to ask, and a model that stops you from asking it is nonsense.
    
   p So: spaCy's entity recognizer is an instance of class #[code spacy.parser.Parser]. It holds an instance of #[code spacy.parser.ner.BiluoPushDown], which owns an array of #[code Transition] structs. Each struct holds three function pointers, that are used to apply the action, determine whether the action is valid given the current state, and determine what its cost would be with respect to a gold-standard analysis.
   
   p Entity recognition proceeds word-by-word. At each word we ask the model to score all the actions. We then loop over the structs and ask each to tell us whether they are valid. We then apply the highest scoring valid action, updating the state in place. At the end of the sentence we apply the analysis to the document.
   
   p We can condition the decisions on the whole document, and on all previous decisions. So we can easily ask questions about what entities have been frequent, and how we previously tagged some word. We don't currently have any features that do this, although research suggests they can be effective. We can also easily ask any questions about the current open entity, including whether it currently matches a gazetteer.

   p The parser's model employs sparse data structures, so we don't have to resize anything if we add an additional class, and start providing examples of it. This is probably not an optimal way to fit the model, but it's often convenient.
  
      pre
          code
              | $ pip install spacy && python -m spacy.en.download all
  
keep keep keep keep replace keep keep keep keep keep
 * Write more notes about spaCy's NER 
website/src/jade/tutorials/bootstrap-ner-word2vec/index.jade