title: Tokenizing Twitter Posts in Lucene
body: <p>My question in a nutshell: Does anyone know of a <em>TwitterAnalyzer</em> or <em>TwitterTokenizer</em> for Lucene?</p>
  
  <p>More detailed version:</p>
  
  <p>I want to index a number of tweets in Lucene and keep the terms like <em>@user</em> or <em>#hashtag</em> intact. StandardTokenizer does not work because it discards the punctuation (but it does other useful stuff like keeping domain names, email addresses or recognizing acronyms). How can I have an analyzer which does everything StandardTokenizer does but does not touch terms like @user and #hashtag?</p>
  
  <p>My current solution is to preprocess the tweet text before feeding it into the analyzer and replace the characters by other alphanumeric strings. For example,</p>
  
  <pre><code>String newText = newText.replaceAll("#", "hashtag");
  newText = newText.replaceAll("@", "addresstag");
  </code></pre>
  
  <p>Unfortunately this method breaks legitimate email addresses but I can live with that. Does that approach make sense?</p>
  
  <p>Thanks in advance!</p>
  
  <p>Amaç</p>