title: error in Naive bayes classifier
body: <p>i'm beginner in machine learning and i'm trying to implement my first Naive Bayes by myself for better understanding. So, i have dataset from <a href="http://archive.ics.uci.edu/ml/datasets/Adult" rel="nofollow">http://archive.ics.uci.edu/ml/datasets/Adult</a> (american census data, classes are '&lt;=50k' and '>50k'). </p>
  
  <p>Here is my python code:</p>
  
  <pre><code>#!/usr/bin/python
  
  import sys
  import csv
  
  words_stats = {} # {'word': {'class1': cnt, 'class2': cnt'}}
  words_cnt = 0
  
  targets_stats = {} # {'class1': 3234, 'class2': 884} how many words in each class
  class_stats = {} # {'class1': 7896, 'class2': 3034} how many lines in each class
  items_cnt = 0
  
  def train(dataset, targets):
      global words_stats, words_cnt, targets_stats, items_cnt, class_stats
  
      num = len(dataset)
      for item in xrange(num):
          class_stats[targets[item]] = class_stats.get(targets[item], 0) + 1
  
          for i in xrange(len(dataset[item])):
              word = dataset[item][i]
              if not words_stats.has_key(word):
                  words_stats[word] = {}
  
              tgt = targets[item]
  
              cnt = words_stats[word].get(tgt, 0)
              words_stats[word][tgt] = cnt + 1
  
              targets_stats[tgt] = targets_stats.get(tgt, 0) + 1
              words_cnt += 1
  
      items_cnt = num
  
  def classify(doc, tgt_set):
      global words_stats, words_cnt, targets_stats, items_cnt
  
      probs = {} #the probability itself P(c|W) = P(W|c) * P(c) / P(W)
      pc = {} #probability of the class in document set P(c)
      pwc = {} #probability of the word set in particular class. P(W|c)
      pw = 1 #probability of the word set in documet set
  
      for word in doc:
          if word not in words_stats:
              continue #dirty, very dirty 
          pw = pw * float(sum(words_stats[word].values())) / words_cnt
  
      for tgt in tgt_set:
          pc[tgt] = class_stats[tgt] / float(items_cnt)
          for word in doc:
              if word not in words_stats:
                  continue #dirty, very dirty
              tgt_wrd_cnt = words_stats[word].get(tgt, 0)
              pwc[tgt] = pwc.get(tgt, 1) * float(tgt_wrd_cnt) / targets_stats[tgt]
  
          probs[tgt] = (pwc[tgt] * pc[tgt]) / pw
  
      l = sorted(probs.items(), key = lambda i: i[1], reverse=True)
      print probs
      return l[0][0]
  
  def check_results(dataset, targets):
      num = len(dataset)
      tgt_set = set(targets)
      correct = 0
      incorrect = 0
  
      for item in xrange(num):
          res = classify(dataset[item], tgt_set)
          if res == targets[item]:
              correct = correct + 1
          else:
              incorrect = incorrect + 1
  
      print 'correct:', float(correct) / num, ' incorrect:', float(incorrect) / num
  
  def load_data(fil):
      data = []
      tgts = []
  
      reader = csv.reader(fil)
      for line in reader:
          d = [x.strip() for x in line]
          if '?' in d:
              continue
  
          if not len(d):
              continue
  
          data.append(d[:-1])
          tgts.append(d[-1:][0])
  
      return data, tgts
  
  if __name__ == '__main__':
      if len(sys.argv) &lt; 3:
          print './program train_data.txt test_data.txt'
          sys.exit(1)
  
      filename = sys.argv[1]
      fil = open(filename, 'r')
      data, tgt = load_data(fil)
      train(data, tgt)
  
      test_file = open(sys.argv[2], 'r')
      test_data, test_tgt = load_data(test_file)
  
      check_results(test_data, tgt)
  </code></pre>
  
  <p>it gives ~61% of correct results. when i print probabilities i get the following:</p>
  
  <pre><code>{'&lt;=50K': 0.07371606889800396, '&gt;50K': 15.325378327213354}
  </code></pre>
  
  <p>but in case of correct classifier i expect to see sum of both probabilities equal to 1.
  At first i thought the problem is in float underflow and tried to make all calculations in logarithms, but results were similiar.
  i understand that omitting some words is gonna affect accuracy, but the probabilities are sooo wrong.</p>
  
  <p>What do i do wrong or don't understand? </p>
  
  <p>for your convinience i've uploaded dataset and python script here:
  <a href="https://dl.dropboxusercontent.com/u/36180992/adult.tar.gz" rel="nofollow">https://dl.dropboxusercontent.com/u/36180992/adult.tar.gz</a></p>
  
  <p>Thank you for your help.</p>