title: Language detection for very short text
body: <p>I'm creating an application for detecting the language of short texts, with an average of &lt; 100 characters and contains slang (e.g tweets, user queries, sms).</p>
  
  <p>All the libraries I tested work well for normal web pages but not for very short text. The library that's giving the best results so far is Chrome's Language Detection (CLD) library which I had to build as a shared library.</p>
  
  <p>CLD fails when the text is made of very short words. After looking at the source code of CLD, I see that it uses 4-grams so that could be the reason.</p>
  
  <p>The approach I'm thinking of right now to improve the accuracy is:</p>
  
  <ul>
  <li>Remove brand names, numbers, urls and words like "software", "download", "internet"</li>
  <li>Use a dictionary When the text contains a number of short words above a threashold or when it contains too few words.</li>
  <li>The dictionary is created from wikipedia news articles + hunspell dictionaries.</li>
  </ul>
  
  <p>What dataset is most suitable for this task? And how can I improve this approach? </p>
  
  <p>So far I'm using EUROPARL and Wikipedia articles. I'm using NLTK for most of the work.</p>