title: How to auto-tag content, algorithms and suggestions needed
body: <p>I am working with some really large databases of newspaper articles, I have them in a MySQL database, and I can query them all.</p>
  
  <p>I am now searching for ways to help me tag these articles with somewhat descriptive tags. </p>
  
  <p>All these articles is accessible from a URL that looks like this:</p>
  
  <pre><code>http://web.site/CATEGORY/this-is-the-title-slug
  </code></pre>
  
  <p>So at least I can use the category to figure what type of content that we are working with. However, I also want to tag based on the article-text.</p>
  
  <p>My initial approach was doing this:</p>
  
  <ol>
  <li>Get all articles</li>
  <li>Get all words, remove all punctuation, split by space, and count them by occurrence </li>
  <li>Analyze them, and filter common non-descriptive words out like "them", "I", "this", "these", "their" etc.</li>
  <li>When all the common words was filtered out, the only thing left is words that is tag-worthy.</li>
  </ol>
  
  <p>But this turned out to be a rather manual task, and not a very pretty or helpful approach. </p>
  
  <p>This also suffered from the problem of words or names that are split by space, for example if 1.000 articles contains the name "John Doe", and 1.000 articles contains the name of "John Hanson", I would only get the word "John" out of it, not his first name, and last name.</p>