title: How to get sentence/text similarity of new corpus from a WIKI-doc2vec model?
body: <p>I am working on a NLP project at the moment. One task is to compare how similar two news articles/titles are. </p>
  
  <p>To do that, i have already trained a doc2vec model using English wikipeida articles on gensim library. Now i want to compare similarities of new text by inferring vectors from the wiki doc2vec model. 
  One of the method i have tried is the gensim docvecs's 'similarity_unseen_docs' function. However, the result isn't really intuitive.
  May i know are there any other way to get a similarity score with better performance? Or maybe some part of code is wrong?  </p>
  
  <p>Training of the doc2vec model:</p>
  
  <pre class="lang-py prettyprint-override"><code>models = Doc2Vec(dm=1, dm_mean=1, vector_size=300, window=5, min_count=20, epochs =10, sample=0.001, negative=5, workers=cores)
  </code></pre>
  
  <p>Comparing similarity:</p>
  
  <pre class="lang-py prettyprint-override"><code>arg1='China is  the leader in manufacturing electric vehicle'
  arg2='The biggest electric car producer is China'
  
  model.docvecs.similarity_unseen_docs(model=wikimodel, doc_words1=tokenize_text(arg1), 
                                           doc_words2=tokenize_text(arg2), alpha=0.025, min_alpha=0.0001, steps=50)
  </code></pre>
  
  <p>The output similarity score is only ~0.25 which doesn't seem to be good.</p>
  
  <p>Here's the code for the training: </p>
  
  <pre class="lang-py prettyprint-override"><code>from gensim.corpora.wikicorpus import WikiCorpus
  from gensim.models.doc2vec import Doc2Vec, TaggedDocument
  from gensim.models import KeyedVectors
  from pprint import pprint
  import multiprocessing
  
  #read the wiki corpus
  wiki = WikiCorpus("trend_analysis/enwiki-latest-pages-articles.xml.bz2")
  
  #define the class to convert wikicorpus into suitable form to train Doc2vec
  class TaggedWikiDocument(object):
      def __init__(self, wiki):
          self.wiki = wiki
          self.wiki.metadata = True
      def __iter__(self):
          for content, (page_id, title) in self.wiki.get_texts():
              yield TaggedDocument([c for c in content], [title])
  
  documents = TaggedWikiDocument(wiki)
  
  
  cores = multiprocessing.cpu_count()
  
  #initialize a model and choose training method. (we use DM here)
  models = Doc2Vec(dm=1, dm_mean=1, vector_size=300, window=5, min_count=20, epochs =10, sample=0.001, negative=5, workers=cores)
  
  #building vocab for the model
  models.build_vocab(documents)
  
  #train the Doc2vec model
  models.train(documents,total_examples=models.corpus_count, epochs=1)
  </code></pre>
  
  <p>Please let me know if the training is having any error that caused the inaccuracy for the inferring of unseen docs.</p>