title: Complication using log-probabilities - Naive Bayes text classifier
body: <p>I'm constructing a Naive Bayes text classifier from scratch in Python and I am aware that, upon encountering a product of very small probabilities, using a logarithm over the probabilities is a good choice.</p>
  
  <p>The issue now, is that the mathematical function that I'm using has a summation OVER a product of these extremely small probabilities.</p>
  
  <p>To be specific, I'm trying to calculate the total word probabilities given a mixture component (class) over all classes.</p>
  
  <p>Just plainly adding up the logs of these total probabilities is incorrect, since the log of a sum is not equal to the sum of logs.</p>
  
  <p>To give an example, lets say that I have 3 classes, 2000 words and 50 documents.
  Then I have a word probability matrix called wordprob with 2000 rows and 3 columns.</p>
  
  <p>The algorithm for the total word probability in this example would look like this:</p>
  
  <pre><code>sum = 0
  for j in range(0,3):
      prob_product = 1
      for i in words:  #just the index of words from my vocabulary in this document
          prob_product = prob_product*wordprob[i,j]
      sum = sum + prob_product
  </code></pre>
  
  <p>What ends up happening is that prob_product becomes 0 on many iterations due to many small probabilities multiplying with each other.</p>
  
  <p>Since I can't easily solve this with logs (because of the summation in front) I'm totally clueless.</p>
  
  <p>Any help will be much appreciated.</p>