title: Parallelize this nested for loop in python
body: <p>I'm struggling again to improve the execution time of this piece of code. Since the calculations are really time-consuming I think that the best solution would be to parallelize the code. </p>
  
  <p>I was first working with maps as explained in <a href="https://stackoverflow.com/questions/29217088/parallelize-a-nested-for-loop-in-python-for-finding-the-max-value">this</a> question, but then I tried a more simple approach thinking that I could find a better solution. However I couldn't come up with anything yet, so since it's a different problem I decided to post it as a new question.</p>
  
  <p>I am working on a Windows platform, using Python 3.4.</p>
  
  <p>Here's the code:</p>
  
  <pre><code>similarity_matrix = [[0 for x in range(word_count)] for x in range(word_count)]
  for i in range(0, word_count):
      for j in range(0, word_count):
          if i &gt; j:
              similarity = calculate_similarity(t_matrix[i], t_matrix[j])
              similarity_matrix[i][j] = similarity
              similarity_matrix[j][i] = similarity
  </code></pre>
  
  <p>This is the <code>calculate_similarity</code> function:</p>
  
  <pre><code>def calculate_similarity(array_word1, array_word2):
        denominator = sum([array_word1[i] + array_word2[i] for i in range(word_count)])
        if denominator == 0:
            return 0
        numerator = sum([2 * min(array_word1[i], array_word2[i]) for i in range(word_count)])
        return numerator / denominator
  </code></pre>
  
  <p>And the explanation for the code:</p>
  
  <ul>
  <li><code>word_count</code> is the total number of unique words stored in a list</li>
  <li><code>t_matrix</code> is a matrix containing a value for each pair of words</li>
  <li>the output should be <code>similarity_matrix</code> whose dimension is <code>word_count x word_count</code> also containing a similarity value for each pair of words</li>
  <li>it's ok to keep both matrices in memory</li>
  <li>after these computations I can easily find the most similar word for each words (or the top three similar words, as the task may require)</li>
  <li><code>calculate_similarity</code> takes two float lists, each for a separate word (each is a row in the t_matrix)</li>
  </ul>
  
  <p>I work with a list of 13k words, and if I calculated correctly the execution time on my system would be a few days. So, anything that will do the job in one day would be wonderful!</p>
  
  <p>Maybe only parellelizing the calculation of <code>numerator</code> and <code>denominator</code> in <code>calculate_similarity</code> would make a significant improvement.</p>