title: Is it possible to share a costly object between multiple Gnu Parallel processes?
body: <p>I am loading a spacy dictionary for the purposes of lemmatizing multiple documents. I am using Gnu Parallel to use my lemmatization script on multiple documents (over 1000 documents) to speed the lemmatization up. However loading the spacy dictionary is a very costly step, which I would ideally like to load once and share across all the processes. Below is the dictionary I am loading. </p>
  
  <p>The following questions ask similar to what I am asking, but there are no conclusive answers. </p>
  
  <p><a href="https://stackoverflow.com/questions/11618543/sharing-shared-object-between-multiple-processes">Sharing shared object between multiple processes</a></p>
  
  <p><a href="https://stackoverflow.com/questions/34933830/would-it-be-possible-to-share-some-memory-with-gnu-parallel">Would it be possible to share some memory with GNU Parallel?</a></p>
  
  <pre><code>nlp = spacy.load('en', disable=['parser', 'ner'])
  
  </code></pre>
  
  <pre><code>ls -d -1 /home/ndg/arc/reddit/2015/RC_2015-[0][1-5]*.gz | parallel -j20 --pipe parallel -j100 --no-notice python lemmatize_subreddit_posts.py
  </code></pre>