Split large log file like a dictionary I have a large log file in the following format. Do note that this is already sorted alphabetically. I wish to split it into smaller pieces, and with file names reflecting the content of the file (first three char of the row).

I wish for the code to be efficient. Speed and simplicity is the key. The log file is in 10s of GB in size, and there are 1000s of such logs. 

Secondly, for large output files (only if the output file is greater than 4 lines), the script should sub-split the file into smaller pieces, using the next alphabet. I am aware this can be done by 2 separate awk commands, but that would not be efficient. Can you pls help me update the existing code?

Input file:

Jon1,details
Jon2,details
Jon4,details
Ron1,detailsaa
Ron1,detailsbb
Ron1,detailscc
Ron1,detailsdd
Ron1,detailsee
Ron2,detailsff


Output I wish for:

cat Jon
Jon1,details
Jon2,details
Jon4,details

cat Ron1
Ron1,detailsaa
Ron1,detailsbb
Ron1,detailscc
Ron1,detailsdd
Ron1,detailsee

cat Ron2
Ron2,detailsff


Current code

awk '
  fname != $1{
    close(fname)
    fname=$1
  }
  {print $0 &gt; fname}' FS="[[:digit:]]+" ./hugeFile
Since you are concerned about execution speed, you should write this program in some other programming languages as well and compare their performance. I'd take Java, C++, Go, Perl, Python and Rust. Traditionally, AWK gets the job done but is not optimized for processing huge amounts of data.

Instead of setting FS on the command line, it should be in a BEGIN clause since it is an essential part of the program:

BEGIN {
    FS = /[[:digit:]]+/;
}


Asking for code that is not yet implemented is off-topic on this site, so I won't comment much on that. The code will become much larger by that requirement though and will require more thought. Be prepared to have a collection of small sample files to test the edge cases automatically.