Extracting emails from log files I have huge number of text files, each several hundred MB in size. Unfortunately, they are not all fully standardized in any one format. Plus there is a lot of legacy in here, and a lot of junk and garbled text. I wish to check all of these files to find rows with a valid email ID, and if it exists then print it to a file named the first-char of the email ID. Hence, multiple text files get parsed and organized into files named a-z and 0-9. In case the email address starts with a special character, then it will get written into a file called "_" underscore. The script also trims the rows to remove whitespaces; and replaces single and double quotes (this is an application requirement)

My script works fine. There are no errors or bugs in it. But it is incredibly slow. My question: is there a more efficient way to achieve this? Parsing 30 GB logs takes me about 12 hrs - way too much! Will grep/cut/sed/another be any faster?

Sample txt File

!bar@foo.com,address
#john@foo.com;address
john@foo.com;address µÖ
email1@foo.com;username;address
email2@foo.com;username
  email3@foo.com,username;address   [spaces at the start of the row]
 email4@foo.com|username|address   [tabs at the start of the row]


My Code:

awk -F'[,|;: \t]+' '{
    gsub(/^[ \t]+|[ \t]+$/, "")
    if (NF&gt;1 &amp;&amp; $1 ~ /^[[:alnum:]_.+-]+@[[:alnum:]_.-]+\.[[:alnum:]]+$/)
    {
        gsub(/"/, "DQUOTES")
        gsub("\047", "SQUOTES")
        r=gensub("[,|;: \t]+",":",1,$0)
        a=tolower(substr(r,1,1))
        if (a ~ /^[[:alnum:]]/)
            print r &gt; a
        else
            print r &gt; "_"
    }
    else
        print $0 &gt; "ErrorFile"
}' *.txt
Don't know if this is affecting your performance, but one thing to check is that you are using an up-to-date version of awk. In particular, the version of awk that ships on MacOS is very slow. You may get dramatic difference by installing the latest GNU gawk. I've measured order of magnitude performance improvements by upgrading. (On MacOS, Homebrew and MacPorts are two popular package managers that can install the latest version.)