title: Multi-Threaded NLP with Spacy pipe
body: <p>I'm trying to apply Spacy NLP (Natural Language Processing) pipline to a big text file like Wikipedia Dump. Here is my code based on Spacy's <a href="https://spacy.io/docs" rel="noreferrer">documentation</a> example:</p>
  
  <pre><code>from spacy.en import English
  
  input = open("big_file.txt")
  big_text= input.read()
  input.close()
  
  nlp= English()    
  
  out = nlp.pipe([unicode(big_text, errors='ignore')], n_threads=-1)
  doc = out.next() 
  </code></pre>
  
  <p>Spacy applies all nlp operations like POS tagging, Lemmatizing and etc all at once. It is like a pipeline for NLP that takes care of everything you need in one step. Applying pipe method tho is supposed to make the process a lot faster by multithreading the expensive parts of the pipeline. But I don't see big improvement in speed and my CPU usage is around 25% (only one of 4 cores working). I also tried to read the file in multiple chuncks and increase the batch of input texts:</p>
  
  <pre><code>out = nlp.pipe([part1, part2, ..., part4], n_threads=-1)
  </code></pre>
  
  <p>but still the same performance. Is there anyway to speed up the process? I suspect that OpenMP feature should be enabled compiling Spacy to utilize multi-threading feature. But there is no instructions on how to do it on Windows. </p>