title: Convert a column in a dask dataframe to a TaggedDocument for Doc2Vec
body: <h1>Intro</h1>
  <p>Currently I am trying to use dask in concert with gensim to do NLP document computation and I'm running into an issue when converting my corpus into a &quot;<a href="https://radimrehurek.com/gensim/models/doc2vec.html#gensim.models.doc2vec.TaggedDocument" rel="nofollow noreferrer">TaggedDocument</a>&quot;.</p>
  <p>Because I've tried so many different ways to wrangle this problem I'll list my attempts.</p>
  <p>Each attempt at dealing with this problem is met with slightly different woes.</p>
  <h1>First some initial givens.</h1>
  <h2>The Data</h2>
  <pre><code>df.info()
  &lt;class 'dask.dataframe.core.DataFrame'&gt;
  Columns: 5 entries, claim_no to litigation
  dtypes: object(2), int64(3)
  </code></pre>
  <pre><code>  claim_no   claim_txt I                                    CL ICC lit
  0 8697278-17 battery comprising interior battery active ele... 106 2 0
  </code></pre>
  <h2>Desired Output</h2>
  <pre><code>&gt;&gt;tagged_document[0]
  &gt;&gt;TaggedDocument(words=['battery', 'comprising', 'interior', 'battery', 'active', 'elements', 'battery', 'cell', 'casing', 'said', 'cell', 'casing', 'comprising', 'first', 'casing', 'element', 'first', 'contact', 'surface', 'second', 'casing', 'element', 'second', 'contact', 'surface', 'wherein', 'assembled', 'position', 'first', 'second', 'contact', 'surfaces', 'contact', 'first', 'second', 'casing', 'elements', 'encase', 'active', 'materials', 'battery', 'cell', 'interior', 'space', 'wherein', 'least', 'one', 'gas', 'tight', 'seal', 'layer', 'arranged', 'first', 'second', 'contact', 'surfaces', 'seal', 'interior', 'space', 'characterized', 'one', 'first', 'second', 'contact', 'surfaces', 'comprises', 'electrically', 'insulating', 'void', 'volume', 'layer', 'first', 'second', 'contact', 'surfaces', 'comprises', 'formable', 'material', 'layer', 'fills', 'voids', 'surface', 'void', 'volume', 'layer', 'hermetically', 'assembled', 'position', 'form', 'seal', 'layer'], tags=['8697278-17'])
  &gt;&gt;len(tagged_document) == len(df['claim_txt'])
  </code></pre>
  <h1>Error Number 1 No Generators Allowed</h1>
  <pre><code>def read_corpus_tag_sub(df,corp='claim_txt',tags=['claim_no']):
      for i, line in enumerate(df[corp]):
          yield gensim.models.doc2vec.TaggedDocument(gensim.utils.simple_preprocess(line), (list(df.loc[i,tags].values)))
  
  tagged_document = df.map_partitions(read_corpus_tag_sub,meta=TaggedDocument)
  tagged_document = tagged_document.compute()
  </code></pre>
  <p>TypeError: Could not serialize object of type generator.</p>
  <p>I found no way of getting around this while still using a generator. A fix for this would be great! As this works perfectly fine for regular pandas.</p>
  <h1>Error Number 2 Only the first element of each partition</h1>
  <pre><code>def read_corpus_tag_sub(df,corp='claim_txt',tags=['claim_no']):
      for i, line in enumerate(df[corp]):
          return gensim.models.doc2vec.TaggedDocument(gensim.utils.simple_preprocess(line), (list(df.loc[i,tags].values)))
  
  tagged_document = df.map_partitions(read_corpus_tag_sub,meta=TaggedDocument)
  tagged_document = tagged_document.compute()
  </code></pre>
  <p>This one is a bit dumb as the function won't iterate (I know) but gives the desired format, but only returns the first row in each partition.</p>
  <h1>Error Number 3 function call hangs with 100% cpu</h1>
  <pre><code>def read_corpus_tag_sub(df,corp='claim_txt',tags=['claim_no']):
      tagged_list = []
      for i, line in enumerate(df[corp]):
          tagged = gensim.models.doc2vec.TaggedDocument(gensim.utils.simple_preprocess(line), (list(df.loc[i,tags].values)))
          tagged_list.append(tagged)
      return tagged_list
  </code></pre>
  <p>Near as I can tell when refactoring the return outside the loop this function hangs builds memory in the dask client and my CPU utilization goes to 100% but no tasks are being computed. Keep in mind I'm calling the function the same way.</p>
  <h1>Pandas Solution</h1>
  <pre><code>def tag_corp(corp,tag):
      return gensim.models.doc2vec.TaggedDocument(gensim.utils.simple_preprocess(corp), ([tag]))
  
  tagged_document = [tag_corp(x,y) for x,y in list(zip(df_smple['claim_txt'],df_smple['claim_no']))]
  </code></pre>
  <p>List comp I haven't time tested this solution</p>
  <h1>Other Pandas Solution</h1>
  <pre><code>tagged_document = list(read_corpus_tag_sub(df))
  </code></pre>
  <p>This solution will chug along pretty much for hours. However I don't have enough memory to juggle this thing when it's done.</p>
  <h1>Conclusion(?)</h1>
  <p>I feel Super lost right now. Here is a list of threads I've looked at. I admit to being really new to dask I've just spent so much time and I feel like I'm on a fools errand.</p>
  <ol>
  <li><a href="https://stackoverflow.com/questions/50862165/creating-a-dask-bag-from-a-generator">Dask Bag from generator</a></li>
  <li><a href="https://medium.com/mindorks/speeding-up-text-pre-processing-using-dask-45cc3ede1366" rel="nofollow noreferrer">Processing Text With Dask</a></li>
  <li><a href="https://gdcoder.com/speed-up-pandas-apply-function-using-dask-or-swifter-tutorial/" rel="nofollow noreferrer">Speed up Pandas apply using Dask</a></li>
  <li><a href="https://stackoverflow.com/questions/45545110/how-do-you-parallelize-apply-on-pandas-dataframes-making-use-of-all-cores-on-o">How do you parallelize apply() on Pandas Dataframes making use of all cores on one machine?</a></li>
  <li><a href="https://stackoverflow.com/questions/31361721/python-dask-dataframe-support-for-trivially-parallelizable-row-apply">python dask DataFrame, support for (trivially parallelizable) row apply?</a></li>
  <li><a href="https://stackoverflow.com/questions/39215617/what-is-map-partitions-doing">What is map_partitions doing?</a></li>
  <li><a href="https://stackoverflow.com/questions/47125665/simple-dask-map-partitions-example">simple dask map_partitions example</a></li>
  <li><a href="https://docs.dask.org/en/latest/dataframe-api.html#dask.dataframe.DataFrame.map_partitions" rel="nofollow noreferrer">The Docs</a></li>
  </ol>