Parse and clean large log files I have the code below, which works successfully, and is used to parse, clean log files (very large in size) and output into smaller sized files. Output filename is the first 2 characters of each line. However, if there is a special character in these 2 characters, then it needs to be replaced with a '_'. This will help ensure there is no illegal character in the filename.
This would take about 12-14 mins to process 1 GB worth of logs (on my laptop). Can this be made faster?
For example, would it help to run this parallel? I am aware I could do }' &quot;$FILE&quot; &amp;. However, I tested and that does not help much. Perhaps AWK itself could run in parallel (the equivalent of  print $0 &gt;&gt; Fpath &amp; )?
Sample log file
email1@foo.com:datahere2     
email2@foo.com:datahere2
email3@foo.com datahere2
email5@foo.com;dtat'ah'ere2 
wrongemailfoo.com
nonascii@row.com;data.is.junk-Œœ
email3@foo.com:datahere2

Expected Output
# cat em 
email1@foo.com:datahere2     
email2@foo.com:datahere2
email3@foo.com:datahere2
email5@foo.com:dtat'ah'ere2 
email3@foo.com:datahere2

# cat errorfile
wrongemailfoo.com
nonascii@row.com;data.is.junk-Œœ

Code:
#/bin/sh
pushd &quot;_test2&quot; &gt; /dev/null
for FILE in *
do
    awk '
    BEGIN {
        FS=&quot;:&quot;
    }
    {
        gsub(/^[ \t&quot;'\'']+|[ \t&quot;'\'']+$/, &quot;&quot;)
        $0=gensub(&quot;[,|;: \t]+&quot;,&quot;:&quot;,1,$0)
        if (NF&gt;1 &amp;&amp; $1 ~ /^[[:alnum:]_.+-]+@[[:alnum:]_.-]+\.[[:alnum:]]+$/ &amp;&amp; $0 ~ /^[\x00-\x7F]*$/)
        {
            Fpath=tolower(substr($1,1,2))
            Fpath=gensub(&quot;[^[:alnum:]]&quot;,&quot;_&quot;,&quot;g&quot;,Fpath)
            print $0 &gt;&gt; Fpath
        }
        else
            print $0 &gt;&gt; &quot;errorfile&quot;
    }' &quot;$FILE&quot;
done
popd &gt; /dev/null
Perhaps switching the outputfile for each line can be avoided by sorting the inputfiles first or with some changes in your script:
# Store the 2-letter filenames
outfile[Fpath];
# Store the highest index for given outputfile
i[Fpath]++;
# Store current line in output array for that file
a[Fpath][i[Fpath]]=$0
# And in the END block print for array per output file
      for (out in outfile) {
         for (j=1;j&lt;=i[out]; j++) {
            print a[out][j] &gt;&gt; out;
         }
      }

This results in
awk '
    BEGIN {
        FS=&quot;:&quot;
    }
    {
        gsub(/^[ \t&quot;'\'']+|[ \t&quot;'\'']+$/, &quot;&quot;)
        $0=gensub(&quot;[,|;: \t]+&quot;,&quot;:&quot;,1,$0)
        if (NF&gt;1 &amp;&amp; $1 ~ /^[[:alnum:]_.+-]+@[[:alnum:]_.-]+\.[[:alnum:]]+$/ &amp;&amp; $0 ~ /^[\x00-\x7F]*$/)
        {
            Fpath=tolower(substr($1,1,2))
            Fpath=gensub(&quot;[^[:alnum:]]&quot;,&quot;_&quot;,&quot;g&quot;,Fpath);
            outfile[Fpath];
            i[Fpath]++;
            a[Fpath][i[Fpath]]=$0
        }
        else
            print $0 &gt;&gt; &quot;errorfile&quot;
    }
   END {
      for (out in outfile) {
         for (j=1;j&lt;=i[out]; j++) {
            print a[out][j] &gt;&gt; out;
         }
      }
   } ' &quot;$FILE&quot;