title: Bash: Base64 encode 1 column in a very large .csv and output to new file
body: <p>I've tried using the code below but the csv file has over 80 million lines (roughly 25gb) and some of the special characters seem to break the echo command. The csv has 2 columns separated by a comma.
  ex:</p>
  
  <pre><code>blah, blah2
  data1,data2
  line3,fd$$#$%T%^Y%&amp;$$B
  somedata,%^&amp;%^&amp;%^&amp;^
  </code></pre>
  
  <p>The goal is to take that second column and base64 is to get ready to import into a sql db. I'm doing a base64 encode on the second column so there's unicode support etc and no character will corrupt the db.
   I'm looking for a more efficient way of doing this that won't break on special chars etc.</p>
  
  <pre><code>awk -F "," '
      {
          "echo "$2" | base64" | getline x
          print $1, x
      }
  ' OFS=',' input.csv &gt; base64.csv
  </code></pre>
  
  <p>Error:</p>
  
  <pre><code>sh: 1: Syntax error: word unexpected (expecting ")") : 
  not foundrf :
  not found201054 : 
  not foundth : 
  not foundz09  
  | base64' (Too many open files)ut.csv FNR=1078) fatal: cannot open pipe `echo q1w2e3r4
  </code></pre>