title: Average of word embeddings with TF-IDF score
body: <p>I have been developing a python script to classify if an article is related to a body text or not. For that I have been using ML (SVM classifier) with some features, including average of word embeddings. </p>
  
  <p>The code for calculating the average of word embeddings between the list of articles and bodies is the following:</p>
  
  <pre><code>word2vec_model = gensim.models.KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)
  setm = set(word2vec_model.index2word)
  
  def avg_feature_vector(words, model, num_features, index2word_set):
          #function to average all words vectors in a given paragraph 
          featureVec = np.zeros((num_features,), dtype="float32")
          nwords = 0
          for word in words:
              if word in index2word_set and word not in stop:
                  try:
                      featureVec = np.add(featureVec, model[word])
                      nwords = nwords+1
                  except:
                      pass
          if(nwords&gt;0):
              featureVec = np.divide(featureVec, nwords)
          return featureVec
  
  def doc_similatiry(headlines, bodies):
      X = []
      docs = []
      for i, (headline, body) in tqdm(enumerate(zip(headlines, bodies))):
          headline_avg_vector = avg_feature_vector(lemmatize_str(clean(headline)).split(), word2vec_model, 300, setm)
          body_avg_vector = avg_feature_vector(lemmatize_str(clean(body)).split(), word2vec_model, 300, setm)
          similarity =  1 - distance.cosine(headline_avg_vector, body_avg_vector)
          X.append(similarity)
      return X, docs
  </code></pre>
  
  <p>It seems like the average word2vec is being calculated correctly. However, it has worse scores than the TF-IDF cosine alone. Therefore, my idea was to group these 2 features, by that means, multiplying the TF-IDF score of each word to the word2vec.</p>
  
  <p>Here is my code to do that:</p>
  
  <pre><code>def avg_feature_vector(words, model, num_features, index2word_set, tfidf_vec, vec_repr, pos):
          #function to average all words vectors in a given paragraph (with tfidf feature)
          featureVec = np.zeros((num_features,), dtype="float32")
          nwords = 0
  
          for word in words:
              if word in index2word_set and word not in stop:
                  try:
                      a = tfidf_vec.vocabulary_[word]
                      featureVec = np.add(featureVec, model[word]) * vec_repr[pos, a]
                      nwords = nwords+1
                  except:
                      pass    
          if(nwords&gt;0):
              featureVec = np.divide(featureVec, nwords)
          return featureVec
  
  def doc_similatiry_with_tfidf(headlines, bodies):
  
      X = []
      docs = []
      for i, (headline, body) in tqdm(enumerate(zip(headlines, bodies))):
          docs.append(lemmatize_str(clean(headline)))
          docs.append(lemmatize_str(clean(body)))
      vectorizer = TfidfVectorizer(norm='l2',min_df=0, use_idf=True, smooth_idf=True, stop_words=stop, sublinear_tf=True)
      sklearn_representation = vectorizer.fit_transform(docs)
  
      for i, (headline, body) in tqdm(enumerate(zip(headlines, bodies))):
          a = (clean(headline))
          headline_avg_vector = avg_feature_vector(nltk.word_tokenize(a), word2vec_model, 300, setm, vectorizer, sklearn_representation, 2*i)
          a = (clean(body))
          body_avg_vector = avg_feature_vector(nltk.word_tokenize(a), word2vec_model, 300, setm, vectorizer, sklearn_representation, 2*i+1)
  
          similarity =  1 - distance.cosine(headline_avg_vector, body_avg_vector)
          X.append(similarity)
  
      return X, docs
  </code></pre>
  
  <p>My problem is that this method is getting awful results and I don't know know if there's some logic that explains that (because in theory it should have better results) or if I'm doing something wrong in my code.</p>
  
  <p>Can anyone help me figure this out? Also, I am open to new solutions to solve this problem.</p>
  
  <p>Note: There are some functions used there which I didn't post the code since I thought they were not necessary. If there is something you don't understand I am here to explain it better.</p>