Parse and clean log files
I think there is quite a lot that could be improved on in your approach.
My main piece of advice is to try and process each line in the data only once, since each line is independent you should be able to do this.
I'm not too familiar with pandas but it seems like there are two main areas of concern.

The section where you clean up the data and filter out all the bad emails, you create a mask by executing two regexs on each line and then read through and make copies of the data frame twice using the mask. At this point you have passed over every line in the data 3 times.

    df = df[0].str.strip(' \t&quot;').str.split('[,|;: \t]+', 1, expand=True).rename(columns={0: 'email', 1: 'data'}) 
    mask = (df.email.str.contains(emailreg, regex=True, na=False)) &amp; (~df.data.str.contains(asciireg, regex=True, na=False))
    df2 = df[~mask].copy()
    df = df[mask].copy()
    df2[['email', 'data']].to_csv(&quot;errorfile&quot;, sep=':', index=False, header=False, mode='a', compression='gzip')
    del df2
    del mask


The second section where you breakdown each email into a different file if it is valid. you go through every line in the dataframe for every possible starting letter, and copy the result over to process again. At this point you have gone through each line in the data about 40 times.

for x in &quot;abcdefghijklmnopqrstuvwxyz0123456789&quot;:
    df2 = df[df.email.str.startswith(x)]
    if (df.email.size &gt; 0):
        df2[['email', 'data']].to_csv(x, sep=':', index=False, header=False, mode='a')

Running cProfile on the code, when it just has to read one file with 6 lines in it produces this: 336691 function calls (328148 primitive calls) in 0.974 seconds. Nearly a second to just read and process 6 lines into different files is not good.
Rather than taking a pandas approach I have just written a pure python script that sketches out an alternative strategy. Doing the same test with cProfile produces 11228 function calls (11045 primitive calls) in 0.038 seconds. It might not fit your needs exactly but you could look at it for ideas about how to tweak your script.
import re
import logging

EMAIL_REGEX = r&quot;^\w+(?:[-+.']\w+)*@\w+(?:[-.]\w+)*\.\w+(?:[-.]\w+)*$&quot;
OUTPUT_FILES = &quot;abcdefghijklmnopqrstuvwxyz0123456789&quot;


def configure_logging():
    &quot;&quot;&quot;
    Configure a logger for each possible email start. 
    &quot;&quot;&quot;

    # TODO - Tweak the handlers, output formats and locations 
    # to suit your needs

    error_handler = logging.FileHandler(&quot;error.log&quot;, mode=&quot;a&quot;)
    error_handler.setLevel(logging.ERROR)
    error_handler.setFormatter(logging.Formatter('%(message)s'))

    for entry in OUTPUT_FILES:
        logger = logging.getLogger(entry)
        handler = logging.FileHandler(f&quot;{entry}.log&quot;, mode=&quot;a&quot;)
        handler.setFormatter(logging.Formatter('%(message)s'))
        handler.setLevel(logging.INFO)
        logger.addHandler(handler)
        logger.addHandler(error_handler)
        logger.setLevel(logging.INFO)
    
def gather_files():
    &quot;&quot;&quot;
    Return all the log files that need to be processed.
    &quot;&quot;&quot;
    # TODO - replace with your own logic to find files.
    return [&quot;test_input.csv&quot;]

def process_log_file(log_file_path):
    &quot;&quot;&quot;
    For each line in the log file, process it once.
    &quot;&quot;&quot;
    with open(log_file_path, &quot;r&quot;) as log_file:
        for line in log_file:
            process_line(line)
                
def process_line(line):
    &quot;&quot;&quot;
    Find the email and user from a line, test if the email is valid. Log the data
    to the appropriate place.
    &quot;&quot;&quot;

    # TODO you may wish to change to logic 
    # to decide if the line is valid or not.

    line = line.strip(' \t&quot;\n')
    data = re.split(r'[,|;: \t]+', line, maxsplit=1)
    logger = logging.getLogger(data[0][0])
    if len(data) == 2 and re.match(EMAIL_REGEX, data[0]):
        logger.info(&quot;:&quot;.join(data))
    else:
        logger.error(line)

def main():
    &quot;&quot;&quot;
    Processes each log file in turn.
    &quot;&quot;&quot;
    for log_file_path in gather_files():
        process_log_file(log_file_path)

if __name__ == &quot;__main__&quot;:
    configure_logging()
    main()

```