title: Is it possible to reliably auto-decode user files to Unicode? [C#]
body: <p>I have a web application that allows users to upload their content for processing.  The processing engine expects UTF8 (and I'm composing XML from multiple users' files), so I need to ensure that I can properly decode the uploaded files.</p>
  
  <p>Since I'd be surprised if any of my users knew their files even <em>were</em> encoded, I have very little hope they'd be able to correctly <em>specify</em> the encoding (decoder) to use.  And so, my application is left with task of detecting before decoding.</p>
  
  <p>This seems like such a universal problem, I'm surprised not to find either a framework capability or general recipe for the solution.  Can it be I'm not searching with meaningful search terms?</p>
  
  <p>I've implemented BOM-aware detection (<a href="http://en.wikipedia.org/wiki/Byte_order_mark" rel="nofollow noreferrer">http://en.wikipedia.org/wiki/Byte_order_mark</a>) but I'm not sure how often files will be uploaded w/o a BOM to indicate encoding, and this isn't useful for most non-UTF files.</p>
  
  <p>My questions boil down to:</p>
  
  <ol>
  <li>Is BOM-aware detection sufficient for the vast majority of files?</li>
  <li>In the case where BOM-detection fails, is it possible to try different decoders <em>and determine if they are "valid"</em>?  (My attempts indicate the answer is "no.")</li>
  <li>Under what circumstances will a "valid" file fail with the C# encoder/decoder framework?</li>
  <li>Is there a repository anywhere that has a multitude of files with various encodings to use for testing?</li>
  <li>While I'm specifically asking about C#/.NET, I'd like to know the answer for Java, Python and other languages for the next time I have to do this.</li>
  </ol>
  
  <p>So far I've found:</p>
  
  <ul>
  <li><strike>A "valid" UTF-16 file with Ctrl-S characters has caused encoding to UTF-8 to throw an exception (Illegal character?)</strike> <em>(That was an XML encoding exception.)</em></li>
  <li>Decoding a valid UTF-16 file with UTF-8 <em>succeeds</em> but gives text with null characters. Huh?</li>
  <li>Currently, I only expect UTF-8, UTF-16 and probably ISO-8859-1 files, but I want the solution to be extensible if possible.</li>
  <li>My existing set of input files isn't nearly broad enough to uncover all the problems that will occur with live files.</li>
  <li>Although the files I'm trying to decode are "text" I think they are often created w/methods that leave garbage characters in the files.  Hence "valid" files may not be "pure".  Oh joy.</li>
  </ul>
  
  <p>Thanks.</p>