title: Maximum Doc2vec similarity between observation and subset at given point in time
body: <p>I have a large dataframe (about 30000 obs) named database_finale. The columns relevant for this post are:</p>
  <ul>
  <li>index1: identifies each observation and is the tag in the doc2vec</li>
  <li>app_date2: is the date of each observation written as an integer</li>
  <li>clean_desc: is the cleaned and tokenized text for the similarity</li>
  <li>snow_pat: is a dummy variable that identifies if the observation belongs to category &quot;snow&quot;.</li>
  </ul>
  <p>I want to create a new column containing the maximum textual similarity of each &quot;non-snow&quot; observation at its date to all the &quot;snow&quot; observations existing up to the date of the focal observation. As the date increases with the focal observations, so do the reference observations in &quot;snow&quot; category, which increase in number.</p>
  <p>After identifying the corpus and training the model, I have tried this code on a subsample of 200 observation and it seems to do the job:</p>
  <pre><code>datab=[]
  #daytuple is a tuple of app_date2 values, index_tuple is a tuple of index1 values
  for i, j in [(i,j) for i in daytuple for j in index_tuple]:
      #here I extract the row of the dataframe of the focal non-snow observation for which I want to find the max similarity to snow
      m=database_finale.loc[(database_finale.index1 ==j ) &amp; (database_finale.app_date2 == i ) &amp; (database_finale[&quot;snow_pat&quot;] !=1)]
      result=m.empty
      #if the extracted line is non-empty
      if result== False:
          #here I extract all the &quot;snow&quot; observations up to day i
          l=database_finale.loc[(database_finale.app_date2 &lt;= i) &amp; (database_finale[&quot;snow_pat&quot;] ==1)]
          result2=l.empty
          #if also this one is non-empty       
          if result2==False:
              #I create a list of the &quot;snow&quot; references
              reference_list1=l[&quot;index1&quot;].tolist()
              #I create dictionaries where to store the similarity score
              most_similars_by_key = {}
              most_similars_by_key_2 = {}
              #model was trained before and corpus was already defined
              for doc in corpus_for_doc2vec:
                  #select the tag of the focal patent
                  if doc.tags[0]==j:
                      #extract from the list of snow up to day i the one that is most similar to the focal observation
                      most_similars_by_key[doc.tags[0]] = model.docvecs.most_similar_to_given(j, reference_list1)
                      #I have the tag of the most similar snow observation to the non-snow focal observation, but not the similarity score, thus I extract the similarity score
                      for key in most_similars_by_key:
                          maxim = most_similars_by_key[key]
                          sim_score = model.docvecs.similarity(key, maxim)
                          most_similars_by_key_2[key] = sim_score
                          print(&quot;prova&quot;+str(prova))
                          prova=prova+1
                          #I merge the database with the most similar observation and the similarity score to the original one and append in a list
                          db1=pd.DataFrame.from_dict(most_similars_by_key, orient='index')
                          db1.reset_index(inplace=True)
                          db1=db1.rename(columns={&quot;index&quot;: &quot;index1&quot;, 0: &quot;most_similar_of_snow&quot;})
                          db2=pd.DataFrame.from_dict(most_similars_by_key_2, orient='index')
                          db2.reset_index(inplace=True)
                          db2=db2.rename(columns={&quot;index&quot;: &quot;index1&quot;, 0: &quot;Similar_doc2vec_desc&quot;})
                          db3=pd.merge(left=database_finale, right= db1, how=&quot;left&quot;, left_on=[&quot;index1&quot;], right_on=[&quot;index1&quot;])
                          db4=pd.merge(left=db3, right= db2, how=&quot;left&quot;, left_on=[&quot;index1&quot;], right_on=[&quot;index1&quot;])
                          datab.append(db4)
                          datab = datab[datab['Similar_doc2vec_desc'].notna()]  
                  else:
                      continue
          else:
              continue
      else:
          continue
  #here I cereate the final DB.
  datab = pd.concat(datab)
  </code></pre>
  <p>As mentioned, this code seems to work, but when applying it to the full 30000 observations, it is extremely slow.
  Can anyone help me to optimize the code in order to speed up the computation?</p>
  <p>I have tried to look into parallelizing the process, but I am not really familiar with this practice and it seems it would require writing this for loop as a function, which I am not sure I have the skills to do.</p>