title: Algorithm to find keywords and keyphrases in a string
body: <p>I need advice or directions on how to write an algorithm which will find <strong>keywords or keyphrases</strong> in a string.</p>
  
  <p>The string contains:</p>
  
  <ul>
  <li>Technical information written in English (GB)</li>
  <li>Words are mostly separated by spaces</li>
  <li>A <strong>keyword does not contain a space</strong> but it may contain a hyphen, apostrophe, colon etc.</li>
  <li>A <strong>keyphrase may contain a space, a comma or other punctuation</strong></li>
  <li>If <strong>two or more keywords appear together</strong> then it is likely a keyphrase e.g. "inverter drive"</li>
  <li>The text also contains HTML but this can be removed beforehand if necessary</li>
  <li>Non-keywords would be words like "and", "the", "we", "see", "look" etc.</li>
  <li>Keywords are case-insensitive e.g. "Inverter" and "inverter" are the same keyword</li>
  </ul>
  
  <p>The algorithm has the following requirements:</p>
  
  <ol>
  <li>Operate in a <strong>batch-processing</strong> scenario e.g. run once or twice a day</li>
  <li>Process strings varying in length from roughly <strong>200 to 7000 characters</strong></li>
  <li><strong>Process 1000 strings in less than 1 hour</strong></li>
  <li>Will execute on a server with moderately good power</li>
  <li>Written in one of the following: <strong>C#, VB.NET, or T-SQL</strong> maybe even F#, Python or Lua etc.</li>
  <li><strong>Does not</strong> rely on a list of predefined keywords or keyphrases</li>
  <li><strong>But can</strong> rely on a list of keyword exclusions e.g. "and", "the", "go" etc.</li>
  <li><strong>Ideally transferable</strong> to other languages e.g. doesn't rely on language-specific features e.g. metaprogramming</li>
  <li>Output a list of keyphrases (descending order of frequency) followed by a list of keywords (descending order of frequency)</li>
  </ol>
  
  <p>It would be extra cool if it could process up to 8000 characters in a matter of seconds, so that it could be run in real-time, but I'm already asking enough!</p>
  
  <p><strong>Just looking for advice and directions:</strong></p>
  
  <ul>
  <li>Should this be regarded as two separate algorithms?</li>
  <li>Are there any established algorithms which I could follow?</li>
  <li>Are my requirements feasible?</li>
  </ul>
  
  <p>Many thanks.</p>
  
  <p>P.S. The strings will be retrieved from a SQL Server 2008 R2 database, so ideally the language would have support for this, if not then it must be able to read/write to STDOUT, a pipe, a stream or a file etc.</p>