title: Python Multiprocessing of NLTK word_tokenizer - function never completes
body: <p>I'm performing natural language processing using NLTK on some fairly large datasets and would like to take advantage of all my processor cores. Seems the multiprocessing module is what I'm after, and when I run the following test code I see all cores are being utilized, but the code never completes. </p>
  
  <p>Executing the same task, without multiprocessing, finishes in approximately one minute. </p>
  
  <p>Python 2.7.11 on debian.</p>
  
  <pre><code>from nltk.tokenize import word_tokenize
  import io
  import time
  import multiprocessing as mp
  
  def open_file(filepath):
      #open and parse file
      file = io.open(filepath, 'rU', encoding='utf-8')
      text = file.read()
      return text
  
  def mp_word_tokenize(text_to_process):
      #word tokenize
      start_time = time.clock()
      pool = mp.Pool(processes=8)
      word_tokens = pool.map(word_tokenize, text_to_process)
      finish_time = time.clock() - start_time
      print 'Finished word_tokenize in [' + str(finish_time) + '] seconds. Generated [' + str(len(word_tokens)) + '] tokens'
      return word_tokens
  
  filepath = "./p40_compiled.txt"
  text = open_file(filepath)
  tokenized_text = mp_word_tokenize(text)
  </code></pre>