title: how to transform csv files of various encoding into utf-8
body: <p>I have 95 small CSV files downloaded from web. Their schemas are supposed to be very similar. I am trying to concatenate them with Python pandas, but when calling <code>pd.read_csv</code>, the various encoding of those files are causing problems, and I am not sure what's the best way to transform them into consistent encoding, e.g. utf-8. The encodings include</p>
  
  <pre><code>ASCII text, with CRLF line terminators
  Little-endian UTF-16 Unicode English text, with CRLF line terminators
  Little-endian UTF-16 Unicode text, with CRLF line terminators
  Little-endian UTF-16 Unicode text, with CRLF, CR line terminators
  UTF-8 Unicode (with BOM) English text, with CRLF line terminators
  UTF-8 Unicode (with BOM) text, with CRLF line terminators
  </code></pre>
  
  <p>The above list is generated with </p>
  
  <pre><code>file -b *.csv | sort | uniq
  </code></pre>