title: How to make element comparison between two lists more efficient? (TF-IDF)
body: <p>I have been learning how TF-IDF is calculated and tried to program it myself. I have solved the problem but I assume that there has to be a more efficient way to calculate the answer. I have tried testing it on quite a bit larger datasets and it seems like this function is causing a major bottleneck</p>
  
  <pre><code>import math
  
  
  def calc_idf(doc_length, normalized_list, terms_with_freqs):  # TODO there must be a way to make this more efficient
      dict_idf = {}
      for i in range(len(normalized_list)):
          dict_calc_idf = []
          for word in normalized_list[i]:
              for x in range(len(terms_with_freqs)):
                  print(terms_with_freqs[x][0])
                  if word[0] == terms_with_freqs[x][0]:
                      dict_calc_idf.append(
                          (word[0], math.log(doc_length / terms_with_freqs[x][1], 10)))
          dict_idf[i] = dict_calc_idf
      return dict_idf
  
  
  normalized_list = {0: [('tom', 0.5), ('earns', 0.5)], 1: [('tom', 0.5), ('castaignede', 0.5)], 2: [('aussie', 0.5), ('mcgrath', 0.5)], 3: [('european', 0.5), ('medal', 0.5)]}
  terms_with_freqs = [('tom', 2), ('earns', 1), ('castaignede', 1), ('aussie', 1), ('mcgrath', 1), ('european', 1), ('medal', 1)]
  doc_length = len(normalized_list)
  
  dict = calc_idf(doc_length, normalized_list, terms_with_freqs)
  
  print(dict)
  </code></pre>