title: Delete duplicated feed titles crawled by webhoseio
body: <p>Got news feeds crawled by webhoseio. Now need to delete the duplicated titles. Below are my codes. Something must be wrong because the output still has duplicated titles. Please help find the problem. Thanks.</p>
  
  <pre><code>count_dup = 0
  for j in range(0,len(feeds)):
  SELECTED_INDEX = j
  feed_sel = feeds[SELECTED_INDEX]
  #print(feed_sel['title'])
  feed_hash = Simhash(str(feed_sel['title']))
  dup_indices = index.get_near_dups(feed_hash)
  #print("Number of duplicates (SimHash): " + str(len(dup_indices)))
  
  for dupi in dup_indices:
      try:
          score = calc_similarity(feed_sel['title'], feeds[int(dupi)]['title'], model_word2vec)
      except:
          score = 0
      if score &gt; 0.85:
          if feeds[int(dupi)]['id'] == j:
              print(feeds[int(dupi)]['id'], feeds[int(dupi)]['title'])
          else:
              feeds.pop(feeds[int(dupi)]['id'] - count_dup)
              count_dup += 1
  </code></pre>