anchor.diff:      |  #[+a("/docs/usage/pos-tagging#rule-based-morphology") rule-based morphological analysis].
   
   +code("Example").
  +    from ..symbols import POS, NOUN, VERB, DET
  +
       TAG_MAP = {
           "NNS":  {POS: NOUN, "Number": "plur"},
           "VBG":  {POS: VERB, "VerbForm": "part", "Tense": "pres", "Aspect": "prog"},
           "DT":   {POS: DET}
       }
   
  -+h(3, "tokenizer-exceptions") Tokenizer exceptions
  -
  -p
  -    |  spaCy's #[+a("/docs/usage/customizing-tokenizer#how-tokenizer-works") tokenization algorithm]
  -    |  lets you deal with whitespace-delimited chunks separately. This makes it
  -    |  easy to define special-case rules, without worrying about how they
  -    |  interact with the rest of the tokenizer. Whenever the key string is
  -    |  matched, the special-case rule is applied, giving the defined sequence of
  -    |  tokens. You can also attach attributes to the subtokens, covered by your
  -    |  special case, such as the subtokens #[code LEMMA] or #[code TAG].
  -
  -p
  -    |  Tokenizer exceptions can be added in the following format:
  -
  -+code("language_data.py").
  -    TOKENIZER_EXCEPTIONS = {
  -        "don't": [
  -            {ORTH: "do", LEMMA: "do"},
  -            {ORTH: "n't", LEMMA: "not", TAG: "RB"}
  -        ]
  -    }
  -
  -p
  -    |  Some exceptions, like certain abbreviations, will always be mapped to a
  -    |  single token containing only an #[code ORTH] property. To make your data
  -    |  less verbose, you can use the helper function #[code strings_to_exc()]
  -    |  with a simple array of strings:
  -
  -+code("Example").
  -    from ..language_data import update_exc, strings_to_exc
  -
  -    ORTH_ONLY = ["a.", "b.", "c."]
  -    converted = strings_to_exc(ORTH_ONLY)
  -    # {"a.": [{ORTH: "a."}], "b.": [{ORTH: "b."}], "c.": [{ORTH: "c."}]}
  -
  -    update_exc(TOKENIZER_EXCEPTIONS, converted)
  -
  -p
  -    |  Unambiguous abbreviations, like month names or locations in English,
  -    |  should be added to #[code TOKENIZER_EXCEPTIONS] with a lemma assigned,
  -    |  for example #[code {ORTH: "Jan.", LEMMA: "January"}].
  -
  -+h(3, "custom-tokenizer-exceptions") Custom tokenizer exceptions
  -
  -p
  -    |  For language-specific tokenizer exceptions, you can use the
  -    |  #[code update_exc()] function to update the existing exceptions with a
  -    |  custom dictionary. This is especially useful for exceptions that follow
  -    |  a consistent pattern. Instead of adding each exception manually, you can
  -    |  write a simple function that returns a dictionary of exceptions.
  -
  -p
  -    |  For example, here's how exceptions for time formats like "1a.m." and
  -    |  "1am" are generated in the English
  -    |  #[+src(gh("spaCy", "spacy/en/language_data.py")) language_data.py]:
  -
  -+code("language_data.py").
  -    from ..language_data import update_exc
  -
  -    def get_time_exc(hours):
  -        exc = {}
  -        for hour in hours:
  -            exc["%da.m." % hour] = [{ORTH: hour}, {ORTH: "a.m."}]
  -            exc["%dp.m." % hour] = [{ORTH: hour}, {ORTH: "p.m."}]
  -            exc["%dam" % hour]   = [{ORTH: hour}, {ORTH: "am", LEMMA: "a.m."}]
  -            exc["%dpm" % hour]   = [{ORTH: hour}, {ORTH: "pm", LEMMA: "p.m."}]
  -        return exc
  -
  -
  -    TOKENIZER_EXCEPTIONS = dict(language_data.TOKENIZER_EXCEPTIONS)
  -
  -    hours = 12
  -    update_exc(TOKENIZER_EXCEPTIONS, get_time_exc(range(1, hours + 1)))
  -
  -+h(3, "utils") Shared utils
  -
  -p
  -    |  The #[code spacy.language_data] package provides constants and functions
  -    |  that can be imported and used across languages.
  -
  -+aside("About spaCy's custom pronoun lemma")
  -    |  Unlike verbs and common nouns, there's no clear base form of a personal
  -    |  pronoun. Should the lemma of "me" be "I", or should we normalize person
  -    |  as well, giving "it" — or maybe "he"? spaCy's solution is to introduce a
  -    |  novel symbol, #[code.u-nowrap -PRON-], which is used as the lemma for
  -    |  all personal pronouns.
  -
  -+table(["Name", "Description"])
  -    +row
  -        +cell #[code PRON_LEMMA]
  -        +cell
  -            |  Special value for pronoun lemmas (#[code &quot;-PRON-&quot;]).
  -
  -    +row
  -        +cell #[code DET_LEMMA]
  -        +cell
  -            |  Special value for determiner lemmas, used in languages with
  -            |  inflected determiners (#[code &quot;-DET-&quot;]).
  -
  -    +row
  -        +cell #[code ENT_ID]
  -        +cell
  -            | Special value for entity IDs (#[code "ent_id"])
  -
  -    +row
  -        +cell #[code update_exc(exc, additions)]
  -        +cell
  -            |  Update an existing dictionary of exceptions #[code exc] with a
  -            |  dictionary of #[code additions].
  -
  -    +row
  -        +cell #[code strings_to_exc(orths)]
  -        +cell
  -            |  Convert an array of strings to a dictionary of exceptions of the
  -            |  format #[code {"string": [{ORTH: "string"}]}].
  -
  -    +row
  -        +cell #[code expand_exc(excs, search, replace)]
  -        +cell
  -            |  Search for a string #[code search] in a dictionary of exceptions
  -            |  #[code excs] and if found, copy the entry and replace
  -            |  #[code search] with #[code replace] in both the key and
  -            |  #[code ORTH] value. Useful to provide exceptions containing
  -            |  different versions of special unicode characters, like
  -            |  #[code &apos;] and #[code ’].
  -
  -p
  -    |  If you've written a custom function that seems like it might be useful
  -    |  for several languages, consider adding it to
  -    |  #[+src(gh("spaCy", "spacy/language_data/util.py")) language_data/util.py]
  -    |  instead of the individual language module.
  -
  -+h(3, "shared-data") Shared language data
  -
  -p
  -    |  Because languages can vary in quite arbitrary ways, spaCy avoids
  -    |  organising the language data into an explicit inheritance hierarchy.
  -    |  Instead, reuseable functions and data are collected as atomic pieces in
  -    |  the #[code spacy.language_data] package.
  -
  -+aside-code("Example").
  -    from ..language_data import update_exc, strings_to_exc
  -    from ..language_data import EMOTICONS
  -
  -    # Add custom emoticons
  -    EMOTICONS = EMOTICONS + ["8===D", ":~)"]
  -
  -    # Add emoticons to tokenizer exceptions
  -    update_exc(TOKENIZER_EXCEPTIONS, strings_to_exc(EMOTICONS))
  -
  -+table(["Name", "Description", "Source"])
  -    +row
  -        +cell #[code EMOTICONS]
  -
  -        +cell
  -            |  Common unicode emoticons without whitespace.
  -
  -        +cell
  -            +src(gh("spaCy", "spacy/language_data/emoticons.py")) emoticons.py
  -
  -    +row
  -        +cell #[code TOKENIZER_PREFIXES]
  -
  -        +cell
  -            |  Regular expressions to match left-attaching tokens and
  -            |  punctuation, e.g. #[code $], #[code (], #[code &quot;]
  -
  -        +cell
  -            +src(gh("spaCy", "spacy/language_data/punctuation.py")) punctuation.py
  -
  -    +row
  -        +cell #[code TOKENIZER_SUFFIXES]
  -
  -        +cell
  -            |  Regular expressions to match right-attaching tokens and
  -            |  punctuation, e.g. #[code %], #[code &#41;], #[code &quot;]
  -
  -        +cell
  -            +src(gh("spaCy", "spacy/language_data/punctuation.py")) punctuation.py
  -
  -    +row
  -        +cell #[code TOKENIZER_INFIXES]
  -
  -        +cell
  -            |  Regular expressions to match token separators, e.g. #[code -]
  -
  -        +cell
  -            +src(gh("spaCy", "spacy/language_data/punctuation.py")) punctuation.py
  -
  -    +row
  -        +cell #[code TAG_MAP]
  -
  -        +cell
  -            |  A tag map keyed by the universal part-of-speech tags to
  -            |  themselves with no morphological features.
  -
  -        +cell
  -            +src(gh("spaCy", "spacy/language_data/tag_map.py")) tag_map.py
  -
  -    +row
  -        +cell #[code ENTITY_RULES]
  -
  -        +cell
  -            |  Patterns for named entities commonly missed by the statistical
  -            | entity recognizer, for use in the rule matcher.
  -
  -        +cell
  -            +src(gh("spaCy", "spacy/language_data/entity_rules.py")) entity_rules.py
  -
  -    +row
  -        +cell #[code FALSE_POSITIVES]
  -
  -        +cell
  -            |  Patterns for phrases commonly mistaken for named entities by the
  -            |  statistical entity recognizer, to use in the rule matcher.
  -
  -        +cell
  -            +src(gh("spaCy", "spacy/language_data/entity_rules.py")) entity_rules.py
  -
  -p
  -    |  Individual languages can extend and override any of these expressions.
  -    |  Often, when a new language is added, you'll find a pattern or symbol
  -    |  that's missing. Even if this pattern or symbol isn't common in other
  -    |  languages, it might be best to add it to the base expressions, unless it
  -    |  has some conflicting interpretation. For instance, we don't expect to
  -    |  see guillemot quotation symbols (#[code &raquo;] and #[code &laquo;]) in
  -    |  English text. But if we do see them, we'd probably prefer the tokenizer
  -    |  to split it off.
  ++h(3, "morph-rules") Morph rules
   
   +h(2, "vocabulary") Building the vocabulary
   
  
positive.code:     |  subpackage of spaCy, named according to the language's
      |  #[+a("https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes") ISO code].
      |  For instance, code and resources specific to Spanish are placed into a
      |  folder #[code spacy/es], which can be imported as #[code spacy.es].
  
  p
      |  To get started, you can use our
  
negative.code:     |  needs to know the language's character set. If the language you're adding
      |  uses non-latin characters, you might need to add the required character
      |  classes to the global
      |  #[+src(gh("spacy", "spacy/lang/punctuation.py")) punctuation.py].
      |  spaCy uses the #[+a("https://pypi.python.org/pypi/regex/") #[code regex] library]
      |  to keep this simple and readable. If the language requires very specific
      |  punctuation rules, you should consider overwriting the default regular
      |  expressions with your own in the language's #[code Defaults].
  
  p
      |  Here's an overview of the individual components that can be included
      |  in the language data. For more details on them, see the sections below.
  
  +table(["File name", "Variables", "Description"])
      +row
          +cell #[+src(gh()) stop_words.py]
          +cell #[code STOP_WORDS] (set)
          +cell
              |  List of most common words. Matching tokens will return #[code True]
              |  for #[code is_stop].
  
      +row
          +cell #[+src(gh()) tokenizer_exceptions.py]
          +cell #[code TOKENIZER_EXCEPTIONS] (dict), #[code TOKEN_MATCH] (regex)
          +cell
              |  Special-case rules for the tokenizer, for example, contractions
              |  and abbreviations containing punctuation.
  
      +row
          +cell #[+src(gh()) punctuation.py]
          +cell
              |  #[code TOKENIZER_PREFIXES], #[code TOKENIZER_SUFFIXES],
              |  #[code TOKENIZER_INFIXES] (dicts)
          +cell Regular expressions for splitting tokens, e.g. on punctuation.
  
      +row
          +cell #[+src(gh()) lex_attrs.py]
          +cell #[code LEX_ATTRS] (dict)
          +cell
              |  Functions for setting lexical attributes on tokens, e.g.
              |  #[code is_punct] or #[code like_num].
  
      +row
          +cell #[+src(gh()) tag_map.py]
          +cell #[code TAG_MAP] (dict)
          +cell
              |  Dictionary mapping strings in your tag set to
              |  #[+a("http://universaldependencies.org/u/pos/all.html") Universal Dependencies]
              |  tags.
  
      +row
          +cell #[+src(gh()) morph_rules.py]
          +cell #[code MORPH_RULES] (dict)
          +cell
  
      +row
          +cell #[+src(gh()) lemmatizer.py]
          +cell #[code LOOKUP] (dict)
          +cell
              |  Lookup-based lemmatization table. If more lemmatizer data is
              |  available, it should live in #[code /lemmatizer/lookup.py].
  
      +row
          +cell /lemmatizer
          +cell #[code LEMMA_RULES], #[code LEMMA_INDEX], #[code LEMMA_EXC] (dicts)
          +cell Lemmatization rules, keyed by part of speech.
  
  +h(3, "stop-words") Stop words
  
  p
      |  A #[+a("https://en.wikipedia.org/wiki/Stop_words") "stop list"] is a
      |  classic trick from the early days of information retrieval when search
      |  was largely about keyword presence and absence. It is still sometimes
      |  useful today to filter out common words from a bag-of-words model. To
      |  improve readability, #[code STOP_WORDS] are separated by spaces and
      |  newlines, and added as a multiline string.
  
  +aside("What does spaCy consider a stop word?")
      |  There's no particularly principal logic behind what words should be
  