title: Efficient way to generate word co-ocurrence numpy matrix in Python and store frequencies to calculate cosine similarity from word vectors
body: <p>I am only starting to code and learn the basic NLP concepts, and I need to create a word co-ocurrence matrix of the Brown corpus with numpy, with a 10-word window. Below is the code that I think works, but I have 2 problems.</p>
  <p>First, it takes over 12 minutes to generate the matrix.
  Second, I need to store the frequencies because I also new to generate a function to calculate the cosine similarity of 2 word vectors. I am very lost at this point, so any help will be appreciated.
  (Sorry, the function definition is not well aligned here, don't know how to fix it).</p>
  <pre><code>def create_co_mat(corpus):
  words_corpus = corpus.words()  # Get all words from corpus
  punctuation = ['!', '&quot;', '#', '$', '%', '&amp;', &quot;'&quot;, '(', ')', '*', '+', ',', '-',
                 '.', '/', ':', ';', '&lt;', '=', '&gt;', '?', '@', '[', '\\', ']', '^', '_', '`', '{', '|', '}', '~', '``',
                 &quot;''&quot;, '--']
  words = [word.lower() for word in words_corpus if word not in punctuation]
  set_words = sorted(set(words))  # 49797 unique words
  words = list(set_words)
  # sentences = []  # &quot;Clean&quot; sentences
  # for sent in corpus.sents():
  #     sentence = [w.lower() for w in sent if w.lower() in words]
  #     sentences.extend(sentence)
  
  co_mat = np.zeros((len(words), len(words)))  # Create a co-occurrence matrix
  # of unique words and initialize them to zero.
  print(&quot;Empty matrix created&quot;)
  
  window_len = 10  # Set window of context words
  # for word in words:
  # Consider each word as a focus word
  for focus_wrd_indx, focus_wrd in enumerate(words):
      # print(focus_wrd_indx, focus_wrd)
      # Get the indices of all the context words for the given focus word
      for contxt_wrd_indx in range((max(0, focus_wrd_indx - window_len)),
                                   (min(len(words), focus_wrd_indx + window_len + 1))):
          if contxt_wrd_indx != focus_wrd_indx:
              # To identify the row number, get the index of the focus_wrd in the uniq_wrds list
              co_mat_row_indx = words.index(focus_wrd)
              # To identify the column number, get the index of the context words in the uniq_wrds list
              co_mat_col_indx = words.index(words[contxt_wrd_indx])
              # Update the respective columns of the corresponding focus word row
              co_mat[co_mat_row_indx][co_mat_col_indx] += 1
  
  print(&quot;Word co-occurrence Matrix with a 10-word window&quot;)
  # np.save('cooc_matrix.npy', co_mat)  # Not save, takes 20GB space from disk!!!
  print(co_mat)
  return co_mat
  </code></pre>