title: How to extract phrases from corpus using gensim
body: <p>For preprocessing the corpus I was planing to extarct common phrases from the corpus, for this I tried using <strong>Phrases</strong> model in gensim, I tried below code but it's not giving me desired output.</p>
  
  <p><strong>My code</strong></p>
  
  <pre><code>from gensim.models import Phrases
  documents = ["the mayor of new york was there", "machine learning can be useful sometimes"]
  
  sentence_stream = [doc.split(" ") for doc in documents]
  bigram = Phrases(sentence_stream)
  sent = [u'the', u'mayor', u'of', u'new', u'york', u'was', u'there']
  print(bigram[sent])
  </code></pre>
  
  <p><strong>Output</strong></p>
  
  <pre><code>[u'the', u'mayor', u'of', u'new', u'york', u'was', u'there']
  </code></pre>
  
  <p><strong>But it should come as</strong> </p>
  
  <pre><code>[u'the', u'mayor', u'of', u'new_york', u'was', u'there']
  </code></pre>
  
  <p>But when I tried to print vocab of train data, I can see bigram, but its not working with test data, where I am going wrong?</p>
  
  <pre><code>print bigram.vocab
  
  defaultdict(&lt;type 'int'&gt;, {'useful': 1, 'was_there': 1, 'learning_can': 1, 'learning': 1, 'of_new': 1, 'can_be': 1, 'mayor': 1, 'there': 1, 'machine': 1, 'new': 1, 'was': 1, 'useful_sometimes': 1, 'be': 1, 'mayor_of': 1, 'york_was': 1, 'york': 1, 'machine_learning': 1, 'the_mayor': 1, 'new_york': 1, 'of': 1, 'sometimes': 1, 'can': 1, 'be_useful': 1, 'the': 1}) 
  </code></pre>