title: Fast way to convert UTF8 to UCS-2 / UTF-16 to feed SSIS Bulk insert task
body: <p>My current situation:</p>
  
  <p>I am using <strong>SSIS 2014</strong> in order to load daily <strong>379 files</strong> (tab delimited files from AS400 in <strong>UTF8</strong>) that take up <strong>20GB</strong> .</p>
  
  <p>My SSIS flow is this:
  <img src="https://i.stack.imgur.com/uPwP9.jpg" alt="ssis flow"></p>
  
  <p>The first task is a script task:</p>
  
  <pre><code>string path= (string)Dts.Variables["dataPath"].Value;
  string name = (string)Dts.Variables["fileName"].Value;
  string from = Path.Combine(path, name) + ".tsv";
  string to = Path.ChangeExtension(from, "txt");
  Dts.Log("Starting "+to.ToUpper(),0,null);
  using (StreamReader reader = new StreamReader(from, Encoding.UTF8, false, 1000000))
  using (StreamWriter writer = new StreamWriter(to,false, Encoding.Unicode, 1000000))
  {
      while (reader.Peek() &gt;= 0)
      {
          writer.WriteLine(reader.ReadLine());    
      }
  }
  </code></pre>
  
  <p>I need to make this flow faster. The only part that i have yet to optimize is the script convert to unicode..</p>
  
  <p>I have tried to disable this task and specify in the Bulk insert task that the file is codepage 650001 , but i get an error that the server does not support this codepage. Obviously SSIS does not conver ton the fly utf8->unicode and then send it to SQL Server.</p>
  
  <p>So i am stuck with the task to convert the files prior to bulk insert.</p>
  
  <p>Now i am searching for the fastest way to do this. The current way is fast but i need to know how i can gain performance in this step with other techniques like:</p>
  
  <p>-Powershell ?</p>
  
  <p>-Command-line application (own built console app or any open-source?)</p>
  
  <p>Thanks everyone </p>