title: Comparing two large lists in python
body: <p>I have one list which contains about 400 words. And another list of lists, in which each list contains about 150,000 words. This list has 20 such lists. </p>
  
  <p>Now I want to see how many of these 400 words appear in all of these 150,000 words list. I also want to know a word from this 400 words, appear how many times in 150k words list, which of these words occur most, how many times etc. </p>
  
  <p>Only solution I can think of is polynomial time solution. It is a very bad solution and will be hell lot slow:</p>
  
  <pre><code>for one_list in list_of_150kwords:
      for key in 400_words:
          for word in one_list:
              if key == word:
                  # count this word
                  # do other stuff
  </code></pre>
  
  <p>This is a very ugly and bad solution, but I can't think of any better. I tried the same with NumPy by converting these lists to NumPy arrays:</p>
  
  <pre><code>list_of_150kwords = numpy.array(list_of_150kwords)
  ...
  </code></pre>
  
  <p>But I still find it very slow. Any other solution? Or any library?</p>