I've a flat file containing about 10 million lines:    query    ID1    content1    content2    query    ID2    content3    content4    ...    content21    query    ID3    content22    content23    ...    content81Any block in the file less than 10 lines should be removed. For example, the first block contains 4 lines (query to content2) and it should be removed. This step need to be done before splitting the blocks into individual files. Any suggestion?
Remove headers and contents from a flat file if they are below a specific line count
sed;awk
Can be done with awk:awk '# define a long blockBEGIN{  long = 10;}# output long block when new block is found($1 == query && n >= long){  print s;}# new block($1 == query){  s = ;  n = 0;}# all lines{  s = (s != ) ? s \n $0 : $0;  n++;}# output the long block if it is the last one in the fileEND{  if (n >= long){    print s;  }}' input.file > output.file