I have a file with multiple columns and have identified lines where specific column values (cols 3-6) have been duplicated using a bash script.Example input:A B C D E F G1 2 T TACA A 3 2 Q3 4 I R 8 2 Q9 3 A C 9 3 P8 3 I R 8 2 QI can display both instances of the repeated values. The other column values (cols 1, 2 and 7+) can be different between the 2 lines hence the need for me to view both instances.I want to save the unique records and the first instance of the duplicated records after sorting these dups have been sorted on col 5 (any order will do) then col 1 (descending order --> largest value first).Desired ouput:A B C D E F G1 2 T TACA A 3 2 Q9 3 A C 9 3 P8 3 I R 8 2 QNB: The ordering on final output is not important as it will be resorted later. Making sure the desired rows are present is what matters.My code so far is:tot=$(awk 'n=x[$3,$6]{print n\n$0;} {x[$3,$6]=$0;}' oldfilename | wc -l)  #counts duplicated records and saves overall count as $totif [ $tot == 0 ] then    awk '{print}' oldfilename >> newfilename  #if no dups found, all lines saved in new fileelse if    awk '(!(n=x[$3,$6]{print n\n$0;} {x[$3,$6]=$0;})' oldfilename >> newfilename  #if dups found, unique lines in old file saved in new fileelse    awk 'n=x[$3,$6]{print n\n$0;} {x[$3,$6]=$0;}' oldfilename > tempfile  #save dups in tempfile    sort -k1,1, -k5,5 tempfile  #sort tempfile on cols 1 then 5 (want descending order)                  fiWhat I am unable to do is take the first instance of each duplicate and save it in newfile and I still have errors in the above code.Please help.
Keeping first instance of duplicates
bash;shell script;text processing;awk;duplicate
sort itself should suffice. First sort such that rows are grouped by field range 3-6, records within each group further ordered by fields 5 and 1. Pipe this to sort -u on 3-6, this disables last-resort comparison and returns the first record from each 3-6 group. Finally, pipe this to sort, this time by fields 5 and 1 sort -k3,6 -k5,5r -k1,1r file | sort -k3,6 -u | sort -k5,5r -k1,1rA B C D E F G1 2 T TACA A 3 2 Q9 3 A C 9 3 P8 3 I R 8 2 Q