title: Use Gensim for scoring features in each document. Also a Python memory issue
body: <p>I am using <code>GENSIM</code> on a corpus of 50000 documents along with a dictionary of around 4000 features. I also have a <code>LSI</code> model already prepared for the same. </p>
  
  <p>Now I want to find the highest matching features for each of the added documents. To find the best features in a particular document, I am running gensim's similarity module for each of the features on all the documents. This gives us a score for each of the feature that we want to use later on. But as you can imagine, this is a costly process as we have to iterate over 50000 indices and run 4000 iterations of similarity on each. </p>
  
  <p>I need a better way of doing this as I run out of 8 GB memory on my system at around 1000 iterations. There's actually no reason for the memory to keep rising as I am only reallocating it during the iterations. Surprisingly the memory starts rising only after around 200 iterations. </p>
  
  <ol>
  <li>Why the memory issue? How can it be solved?</li>
  <li>Is there a better way of finding the highest scored features in a particular document (not topics)?</li>
  </ol>
  
  <p>Here's a snippet of the code that runs out of memory:</p>
  
  <pre><code>dictionary = corpora.Dictionary.load('features-dict.dict')
  corpus = corpora.MmCorpus('corpus.mm')
  lsi = models.LsiModel.load('model.lsi')
  corpus_lsi = lsi[corpus]
  index = similarities.MatrixSimilarity(list(corpus_lsi))
  newDict = dict()
  
  for feature in dictionary.token2id.keys():
    vec_bow = dictionary.doc2bow([feature])
    vec_lsi = lsi[vec_bow]
    sims = index[vec_lsi]
    li = sorted(enumerate(sims * 100), key=lambda item: -item[1])
  
    for data in li:
      dict[data[0]] = (feature,data[1]) # Store feature and score for each document
  
  
  # Do something with the dict created above
  </code></pre>
  
  <p>EDIT: </p>
  
  <p>The memory issue was resolved using a memory profiler. There was something else in that loop that caused it to rise drastically.</p>
  
  <p>Let me explain the purpose in detail. Imagine we are dealing with various recipes (each recipe is document) and each item in our dictionary is an ingredient. Find six such recipes below.</p>
  
  <p><code>corpus = [[Olive Oil, Tomato, Brocolli, Oregano], [Garlic, Olive Oil, Bread, Cheese, Oregano], [Avocado, Beans, Cheese, Lime], [Jalepeneo, Lime, Tomato, Tortilla, Sour Cream], [Chili Sauce, Vinegar, Mushrooms, Rice], [Soy Sauce, Noodles, Brocolli, Ginger, Vinegar]]</code></p>
  
  <p>There are thousands of such recipes. What I am trying to achieve is to assign a weight between 0 and 100 to each of the ingredient (where higher weighted ingredient is the most important or most unique). What would be the best way to achieve this.</p>