title: saving and loading multiple shards made by gensim similarity model
body: <p>My data has more than 1 million rows and while training gensim similarity model, it is making multiple .sav files (model.sav, model.sav.0, model.sav.1 and so on..). Problem is while loading, it is loading only one sub-part, instead of all the sub-parts, hence performing horribly in prediction. Parameters/options are not working as per gensim documentation.</p>
  
  <p>As per the gensim documentation - <a href="https://radimrehurek.com/gensim/similarities/docsim.html" rel="nofollow noreferrer">https://radimrehurek.com/gensim/similarities/docsim.html</a>
  Saving as file handle and giving the following params should have worked - : </p>
  
  <ol>
  <li>model.save(fname_or_handle, separately = None)</li>
  <li>model.load(filepath, mmap = 'r')</li>
  </ol>
  
  <p>Even tried to -</p>
  
  <ol>
  <li>pickle the .sav files ( this pickles the 1st shard only i.e. model.sav)</li>
  <li>compressing all sub-parts as .gz file ( this compresses one shard only , not all the sub-parts) and also gives some sort of pickle error.</li>
  </ol>
  
  <pre><code>tf_idf = gensim.models.TfidfModel(corpus)
  
  sims = gensim.similarities.Similarity('./models/model.sav',tf_idf[corpus],
                                            num_features=len(dictionary))
  sims.save('./models/model.sav')
  
  sims1 = gensim.similarities.Similarity.load(./models/model.sav)
  </code></pre>
  
  <p>Expected results should give all matching documents from corpus, but this gives only from model.sav (the file mentioned while loading). It does NOT even execute the other shards. I checked result from each shard.</p>
  
  <p>Question: How do I use all the sub-files of gensim model to predict similarity of my test document, WITHOUT looping through every sub-file individually and then presenting union of those results.</p>