title: Trouble Parsing Unicode CSV File
body: <p>I'm trying to load a CSV into a DataTable using this:</p>
  
  <pre><code>class CSVReader
  {
      public System.Data.DataTable GetDataTable(string strFileName)
      {
          System.Data.OleDb.OleDbConnection conn = new System.Data.OleDb.OleDbConnection
          (
          "Provider=Microsoft.Jet.OleDb.4.0; Data Source = " + System.IO.Path.GetDirectoryName(strFileName) + 
          "; Extended Properties = \"Text;HDR=YES;FMT=Delimited\""
          );
          conn.Open();
          string strQuery = "SELECT * FROM [" + System.IO.Path.GetFileName(strFileName) + "]";
          System.Data.OleDb.OleDbDataAdapter adapter = new System.Data.OleDb.OleDbDataAdapter(strQuery, conn);
          System.Data.DataSet ds = new System.Data.DataSet("CSV File");
          adapter.Fill(ds);
          return ds.Tables[0];
      }
  }
  </code></pre>
  
  <p>It works fine one one CSV, but not another. Here is the snippet of the file that doesn't load properly:
  <img src="https://i.stack.imgur.com/0RCAO.jpg" alt="enter image description here"></p>
  
  <p>It simply loads "T" as the first column name, and everything else is blank/null. I tried manually looking at the first line with </p>
  
  <pre><code>Using (StreamReader x = new StreamReader(fileName) { string firstline x = x.ReadLine(); }
  </code></pre>
  
  <p>and the equivalent File.ReadAllLines and referenced the array's "0" entry ( file[0] ).</p>
  
  <p>Both simply return "T" as the "first line" and anything beyond that is blank. Any ideas why it only sees the first character in the CSV and nothing else? </p>
  
  <p>EDIT: First line looks like this:</p>
  
  <blockquote>
    <p>TERM(s),OBJECTID,FILE,PATH,HIT COUNT</p>
  </blockquote>
  
  <p>The second line looks like this:</p>
  
  <blockquote>
    <p>"(test)","172911","16369318","Item001.E01/Partition 1/NONAME [NTFS]/[unallocated space]/13621367/16369318","4"</p>
  </blockquote>
  
  <p>EDIT 2:
  I switched over the the lib somebody linked (CVSReader) and it seems to have taken a lot of the pain out. I tried encoding the file in UTF-8 with Notepad++ and it gets farther along, until it gets to:</p>
  
  <blockquote>
    <p>LumenWorks.Framework.IO.Csv.MalformedCsvException was unhandled
      Message=The CSV appears to be corrupt near record '1373' field '3 at position '2601'. Current raw data : '32/System.ServiceModel/06d6eab93282d2b136a377bd50b7c5a9/System.ServiceModel.ni.dll","11"
    "(vc)","40656","Adobe AIR Application Installer.swf","Item001.E01/Partition 1/NONAME [NTFS]/[root]/Program Files/Common Files/Adobe AIR/Versions/1.0/Adobe AIR Application Installer.swf","11"
    "(vc)","503322","䄳䆷䞫䄦䠥","Item001.E01/Partition 1/NONAME [NTFS]/[root]/WINDOWS/Installer/520ae67.msp/䄳䆷䞫䄦䠥","11"</p>
  </blockquote>
  
  <p>I'm guessing that it is taking issue with the foreign characters in UTF-8 encoding. If I leave the file the way it was, original encoding, it processes poorly/incorrectly. I don't want to make the user have to open the file and save it as ASCII/UTF-16 since it is ~90mb. I've been trying to google around, but most people say .NET can handle any encoding.</p>
  
  <p>It seems the file is outputted as UCS-2 LE (which I think is UTF-16, right?). I"m confused why CVSReader/StreamReader are taking issue.</p>
  
  <p><strong>Diagnosed but not fully solved</strong></p>
  
  <p>When I pass a "characterset=Unicode" appended to the string on my OldeDB function it seems to work for USC-2LE/Unicode encoding. I would prefer to use the CSVReader custom lib, but it seems to use TextReader (which as far as I can tell can't handle Unicode). </p>
  
  <p><a href="http://www.codeproject.com/KB/database/CsvReader.aspx" rel="nofollow noreferrer">http://www.codeproject.com/KB/database/CsvReader.aspx</a></p>
  
  <p>THe following code will not work. It doesn't throw an error, but it seems to stall out even on its own thread:</p>
  
  <p>Bad Code for USC2/Unicode:</p>
  
  <pre><code>                using (CsvReader csv = new CsvReader(
                             new StreamReader(kwfile, Encoding.Unicode), true))
              {
                  csv.MissingFieldAction = MissingFieldAction.ReplaceByEmpty;
                  keywordHits.Load(csv);
              }
  </code></pre>
  
  <p>Working, but not preferred solution:</p>
  
  <pre><code>        public System.Data.DataTable GetDataTable(string strFileName)
      {
          System.Data.OleDb.OleDbConnection conn = new System.Data.OleDb.OleDbConnection
          (
          "Provider=Microsoft.Jet.OleDb.4.0; Data Source = " + System.IO.Path.GetDirectoryName(strFileName) +
          "; Extended Properties = \"Text;characterset=Unicode;HDR=YES;FMT=Delimited\""
          );
          conn.Open();
          string strQuery = "SELECT * FROM [" + System.IO.Path.GetFileName(strFileName) + "]";
          System.Data.OleDb.OleDbDataAdapter adapter = new System.Data.OleDb.OleDbDataAdapter(strQuery, conn);
          System.Data.DataSet ds = new System.Data.DataSet("CSV File");
          adapter.Fill(ds);
          return ds.Tables[0];
      }
  </code></pre>
  
  <p>I've tried it with the optional third parameter in the CSVReader as well, and no difference. The program "works" by loading one CSV using the CSVReader class but has to use the OldeDB on the Unicode CSV. Obviously, using StreamReader with the Encoding.Unicode parameter works, but I'd have to re-invent the wheel in parsing out possibly malformed entries. Any thoughts? Or is this the best I can do without rewriting the CSVReader?</p>