title: Ruby CSV library read from file: Determining source file encoding to be provided to foreach method
body: <p>I was facing a problem regarding parsing a CSV file posted here: <a href="https://stackoverflow.com/q/16772830/936494">Ruby unable to parse a CSV file: CSV::MalformedCSVError (Illegal quoting in line 1.)</a></p>
  
  <p>The problem was resolved with the guidance received from @Anand.Posting here the answer for reference in case it helps others.</p>
  
  <p>My file(/tmp/my_data.csv) used in sample code shown in above referred post actually contained the Unicode characters named BOM (Byte Order Mark) at the start of the file .I received the file from a client so don't know how they got in there.</p>
  
  <p>Using @Anand's suggestion of using following code:</p>
  
  <p>in case of string</p>
  
  <pre><code>sub!(/^\xEF\xBB\xBF/, '') 
  </code></pre>
  
  <p>or in case of a file starting with BOM characters worked</p>
  
  <pre><code>CSV.foreach("test.csv", encoding: "bom|utf-8")
  </code></pre>
  
  <p>worked.</p>
  
  <p>However this raises another question that is there a way I can detect problems in CSV file like containing specific encoded characters which is different from standard Unicode? In other words my file contained BOM unicode characters.Could those be detected earlier and fixed before I start reading the file? Is there a way out? I mean the solution shown above required the encoding the source file was in and the target encoding.So there should be a way to determine the source encoding or such characters being present in the source file to be read.If anybody have any idea on this please provide your inputs.</p>
  
  <p>Thanks,
  Jignesh</p>