title: Using NLTK to perform document classification on website content issues with BeautifulSoup and NaiveBayes
body: <p>I have a Python 2.7 project where I want to classify websites based on their content. I have a database in which I numerous website URLs and their associated category. There are many categories (= labels), and I wish to classify new sites into the corresponding category based on their content. I've been following the NLTK classification tutorial/example listed <a href="http://www.nltk.org/book/ch06.html" rel="nofollow">here</a>, but have run into some problems I cannot explain.</p>
  
  <p>Here is an outline of the process that I use:</p>
  
  <ol>
  <li>Use MySQLdb to retrieve the category associated with a given website URL. 
     This will be used when extracting data (content) from the URL to pair it with the
     category (= label) of the site.</li>
  <li>Use a getSiteContent(site) function to extract the content from a website</li>
  </ol>
  
  <p>The above function looks like this:</p>
  
  <pre><code>def getSiteContent(site):
      try:
          response = urllib2.urlopen(site, timeout = 1)
          htmlSource = response.read()
      except Exception as e: # &lt;=== some websites may be inaccessible as list isn't up-to-date
          global errors
          errors += 1
          return ''
  
      soup = BeautifulSoup(htmlSource)
      for script in soup.find_all('script'):
          script.extract()
  
      commonWords = set(stopwords.words('english'))
      commonWords.update(['function', 'document', 'window', 'functions',     'getElementsByTagName', 'parentNode', 'getDocumentById', 'javascript', 'createElement',     'Copyright', 'Copyrights', 'respective', 'owners', 'Contact Us', 'Mobile Version', 'FAQ',     'Privacy Policy', 'Terms of Service', 'Legal Disclaimer' ])
  
      text = soup.get_text()
  
      # Remove ',', '/', '%', ':'
      re.sub(r'(\\d+[,/%:]?\\d*)', '', text)
      # Remove digits
      re.sub(r'\d+', '', text)
      # Remove non-ASCII
      re.sub(r'[^\x00-\x7F]',' ', text)
      # Remove stopwords
      for word in commonWords :
          text = text.replace(' '+word+' ', ' ')
  
      # Tokenize the site content using NLTK
      tokens = word_tokenize(text)
  
      # We collect some word statistics, i.e. how many times a given word appears in the     text
      counts = defaultdict(int)
      for token in tokens:
          counts[token] += 1
  
      features = {}
      # Get rid of words that appear less than 3 times
      for word in tokens:
          if counts[word] &gt;= 3 :
              features['count(%s)' % word] = counts[word]
  
      return features 
  </code></pre>
  
  <p>When all the above is done, I do the following:</p>
  
  <pre><code>train = getTrainingSet(n)
  random.shuffle(train)
  </code></pre>
  
  <p>Where n is the number of sites I wish to train my model against. </p>
  
  <p>Afterwards, I do:</p>
  
  <pre><code>feature_set = []
  count = 0
  for (site, category) in train:
      result = getSiteContent(site)
      count += 1
      if result != '':
          print "%d. Got content for %s" % (count, site)
          feature_set.append((result, category))
      else  :
          print "%d. Failed to get content for %s" % (count, site)
  </code></pre>
  
  <p>The print statements are mainly for debugging purposes at this time. After I do the above, <code>feature_set</code> contains something similar to the following:</p>
  
  <pre><code>print feature_set
  [({u'count(import)': 22, u'count(maxim)': 22, u'count(Maxim)': 5, u'count(css)': 22, u'count(//www)': 22, u'count(;)': 22, u'count(url)': 22, u'count(Gift)': 3, u"count('')": 44, u'count(http)': 22, u'count(&amp;)': 3, u'count(ng16ub)': 22, u'count(STYLEThe)': 3, u'count(com/modules/system/system)': 4, u'count(@)': 22, u'count(?)': 22}, 'Arts &amp; Entertainment'), ({u'count(import)': 3, u'count(css)': 3, u'count(\u05d4\u05d9\u05d5\u05dd)': 4, u'count(\u05de\u05d9\u05dc\u05d5\u05df)': 6, u'count(;)': 3, u'count(\u05e2\u05d1\u05e8\u05d9)': 4, u'count(\u05d0\u05ea)': 3, u'count(\u05de\u05d5\u05e8\u05e4\u05d9\u05e7\u05e1)': 6, u"count('')": 6, u'count(\u05d4\u05d5\u05d0)': 3, u'count(\u05e8\u05d1\u05de\u05d9\u05dc\u05d9\u05dd)': 3, u'count(ver=01122014_4)': 3, u'count(|)': 4, u'count(``)': 4, u'count(@)': 3, u'count(?)': 7}, 'Miscellaneous')]
  </code></pre>
  
  <p>Afterwards, I try to train my classifier and then run it against the test data that I extract from <code>feature_set</code></p>
  
  <pre><code>train_set, test_set = feature_set[len(train)/2:], feature_set[:len(train)/2]
  print "Num in train_set: %d" % len(train_set)
  print "Num in test_set: %d" % len(test_set)
  classifier = nltk.NaiveBayesClassifier.train(train_set) # &lt;=== classified declared on train_set
  print classifier.show_most_informative_features(5)
  print "=== Classifying a site ==="
  print classifier.classify(getSiteContent("http://www.mangaspoiler.com"))
  print "Non-working sites: %d" % errors
  print "Classifier accuracy: %d" % nltk.classify.accuracy(classifier, test_set)
  </code></pre>
  
  <p>This is pretty much exactly how the tutorial on the NLTK documentation website does it. However, the results are the following (given a set of 100 websites):</p>
  
  <pre><code>$ python classify.py
  Num in train_set: 23
  Num in test_set: 50
  Most Informative Features
              count(Pizza) = None           Arts &amp; : Techno =      1.0 : 1.0
  None
  === Classifying a site ===
  Technology &amp; Computing
  Non-working sites: 27
  Classifier accuracy: 0
  </code></pre>
  
  <hr>
  
  <p>Now, there are obviously a few problems with this:</p>
  
  <ol>
  <li><p>The word tokens contain unicode characters such as <code>\u05e2\u05d1\u05e8\u05d9</code>, as it seems that the regex for removing them only works if they are standalone. This is a minor problem.</p></li>
  <li><p>A bigger problem is that even when I <code>print</code> the <code>feature_set</code>, the word tokens are displayed as <code>u'count(...)' = #</code> as opposed to <code>'count(...)' = #</code>. I think this may be a bigger issue and part of why my classifier is failing.</p></li>
  <li><p>The classifier is, obviously, failing catastrophically as some point. The accuracy is listed as <code>0</code> even if I feed my entire dataset into the classifier, which seems extremely unlikely.</p></li>
  <li><p>The <code>Most Informative Features</code> function says that <code>count(Pizza) = None</code>. The code where I declare <code>defaultdict(int)</code>, however, requires that every entry be associated with the number of appearances in the text.</p></li>
  </ol>
  
  <hr>
  
  <p>I am at quite a loss as to why this happens. As far as I can tell, my data is structured identically to the data that the NLTK documentation uses in its tutorial on the website I linked at the top of this question. If anyone who has worked with NLTK has seen this behaviour before, I would greatly appreciate any tips as to what I could be doing wrong.</p>