anchor.diff:      |  loaded. By default, the command expects to be able to find your language
       |  class using #[code spacy.util.get_lang_class(lang_id)].
   
  -+h(3, "word-frequencies") Word frequencies
  -
  -p
  -    |  The #[+a("/docs/usage/cli#model") #[code model] command] expects a
  -    |  tab-separated word frequencies file with three columns:
  -
  -+list("numbers")
  -    +item The number of times the word occurred in your language sample.
  -    +item The number of distinct documents the word occurred in.
  -    +item The word itself.
  -
  -p
  -    |  You should make sure you use the spaCy tokenizer for your
  -    |  language to segment the text for your word frequencies. This will ensure
  -    |  that the frequencies refer to the same segmentation standards you'll be
  -    |  using at run-time. For instance, spaCy's English tokenizer segments
  -    |  "can't" into two tokens. If we segmented the text by whitespace to
  -    |  produce the frequency counts, we'll have incorrect frequency counts for
  -    |  the tokens "ca" and "n't".
  -
  -+h(3, "brown-clusters") Training the Brown clusters
  -
  -p
  -    |  spaCy's tagger, parser and entity recognizer are designed to use
  -    |  distributional similarity features provided by the
  -    |  #[+a("https://github.com/percyliang/brown-cluster") Brown clustering algorithm].
  -    |  You should train a model with between 500 and 1000 clusters. A minimum
  -    |  frequency threshold of 10 usually works well.
  -
  -+h(3, "word-vectors") Training the word vectors
  -
  -p
  -    |  #[+a("https://en.wikipedia.org/wiki/Word2vec") Word2vec] and related
  -    |  algorithms let you train useful word similarity models from unlabelled
  -    |  text. This is a key part of using
  -    |  #[+a("/docs/usage/deep-learning") deep learning] for NLP with limited
  -    |  labelled data. The vectors are also useful by themselves – they power
  -    |  the #[code .similarity()] methods in spaCy. For best results, you should
  -    |  pre-process the text with spaCy before training the Word2vec model. This
  -    |  ensures your tokenization will match.
  -
  -p
  -    | You can use our
  -    |  #[+src(gh("spacy-dev-resources", "training/word_vectors.py")) word vectors training script],
  -    |  which pre-processes the text with your language-specific tokenizer and
  -    |  trains the model using #[+a("https://radimrehurek.com/gensim/") Gensim].
  +
  ++h(2, "train-tagger-parser") Training the tagger and parser
  +
  +p
  +    |  You can now train the model using a corpus for your language annotated
  +    |  with #[+a("http://universaldependencies.org/") Universal Dependencies].
  +    |  If your corpus uses the connlu format, you can use the
  +    |  #[+a("/docs/usage/cli#convert") #[code convert] command] to convert it to
  +    |  spaCy's #[+a("/docs/api/annotation#json-input") JSON format] for training.
  +
  +p
  +    |  Once you have your UD corpus transformed into JSON, you can train your
  +    |  model use the using spaCy's
  +    |  #[+a("/docs/usage/cli#train") #[code train] command]:
  +
  ++code(false, "bash").
  +    python -m spacy train [lang] [output_dir] [train_data] [dev_data] [--n_iter] [--parser_L1] [--no_tagger] [--no_parser] [--no_ner]
  
positive.code:     |  by linear models, while the word vectors are useful for lexical
      |  similarity models and deep learning.
  
  p
      |  Once you've collected the word frequencies, Brown clusters and word
      |  vectors files, you can use the
      |  #[+src(gh("spacy-dev-resources", "training/init.py")) init.py]
      |  script from our
      |  #[+a(gh("spacy-dev-resources")) developer resources], or use the new
      |  #[+a("/docs/usage/cli#model") #[code model] command] to create a data
      |  directory:
  
  
negative.code:         |  #[a(href="#brown-clusters") Brown clusters] and
          |  #[a(href="#word-vectors") word vectors].
  
      +item
          |  #[strong Set up] a #[a(href="#model-directory") model direcory] and #[strong train] the #[a(href="#train-tagger-parser") tagger and parser].
  
  p
      |  For some languages, you may also want to develop a solution for
      |  lemmatization and morphological analysis.
  
  