title: Tokenize text for search in Python
body: <p><strong>Looking for strategies on how to tokenize text for search, and some ideas on how to implement them.</strong>  </p>
  
  <p>Specifically, we are trying to tokenize user generated business reviews to help with our business search engine.  All the code is Python.</p>
  
  <p>I think we need to do at least the following:</p>
  
  <ul>
  <li><p><strong>Convert plural nouns to singulars</strong><br>
  I found a library called <a href="http://pypi.python.org/pypi/inflect" rel="nofollow">inflect</a> that seems to do this well, does anyone have any experience with it?</p></li>
  <li><p><strong>Get rid of all non alpha-numeric characters</strong><br>
  This seems like a job for regex to me, but I'd love to hear any other suggestions</p></li>
  <li><p><strong>Tokenize based on whitespace, converting consecutive whitespace into a single whitespace</strong><br>
  I think this is doable with some custom string manipulation in Python, but there may be a better way.  </p></li>
  </ul>
  
  <p>Does anyone have any other ideas about things I'd need to do to tokenize the text?  Also, what are your thoughts on the techniques and tools mentioned for implementing the strategies above?</p>
  
  <p><strong>Background info</strong>: (from comments to <code>Dough T</code>'s suggestion about Solr or Elastic search)<br>
  We are using <strong><em>ElasticSearch</em></strong>, and we use its tools for basic tokenization. We want to do the tokenization described above separately because, after tokenization, we are going to need to apply some pretty involved semantic analysis to extract meaning from the text. We want the flexibility to tokenize <em>exactly</em> how we specify, and the convenience of having the tokens stored in our own format with our own data annotations attached to them. <br>
  One thing that we absolutely need is a single (large) database record for each token, accessible and modifiable on the fly, with everything relevant about that token's usage in it. I think that rules out just using ES tokenization to process them as the documents get indexed. We could maybe use the ES's analysis module to analyze the text without indexing it, then process each token individually in order to build/update the token's database record... We seek suggestions about this approach.</p>