title: Preserve Line Feeds and Carriage Returns Powershell
body: <p>I have a csv file originated from Linux Server, so they contains <strong>\n</strong> to mark the ending of a line. Now I process this file through a powershell script and replace it's 'text qualifier' to something else (that I do because I am using SSIS to upload the csv feed to database and for some odd reasons SSIS don't supports 'embedded text qualifiers')</p>
  
  <p>Part of the script which do this replacement looks like this</p>
  
  <pre><code>gc $file.FullName |
      % { if($_.indexOf("|~|") -eq -1) {$_ -replace "`"((?:`"`"|.)*?)`"(?!`")", "|~|`$1|~|" -replace "`"`"", "`""} else {$_ -replace " ", " "}} |
      sc  $temppath
  </code></pre>
  
  <p>This scripts works fine but that also change the line feed at the end to <strong>\r\n</strong> That I understand should not have been that big of problem until I realized that my original feed also contains occasional <strong>\r</strong> in the description column which is also getting replaced with <strong>"\r\n"</strong> Now SSIS package is unable to identify where' the csv line ends.</p>
  
  <p>I searched and found this is due to Get-Content which work line by line so I changed the command to following.</p>
  
  <pre><code>[System.IO.File]::ReadAllText($file.FullName) |
              % { if($_.indexOf("|~|") -eq -1) {$_ -replace "`"((?:`"`"|.)*?)`"(?!`")", "|~|`$1|~|" -replace "`"`"", "`""} else {$_ -replace " ", " "}} |
              sc  $temppath
  </code></pre>
  
  <p><strong>That seems to solve my issue but now I am trapped with *"OutOfMemoryException" as some of the csv files are big (about 400-500 MB) *</strong></p>
  
  <p>Any Suggestions what I can possibily do? Perhaps some replacement for ReadAllText() that works for big files?</p>