I have a massive log file that I need to filter. In  this log, I want to display all logs that contain the string dns and I want see them just once.i.e from this:Dec  9 07:24:02 94.15.218.140 syslog: ssk:548.049:is_dns_hijack:1451:isDnsHijack=0Dec  9 07:24:10 90.192.172.112 syslog: ssk:363.217:cmsLck_acquireLockWithTimeoutTraced:98:acquired lock. callerFuncName is_dns_hijack; timeout 12000 millisecondsDec  9 07:24:10 90.192.172.112 syslog: ssk:363.218:cmsLck_releaseLockTraced:144:lock hold time=0ms, acquiring lock callerFuncName is_dns_hijack; releasing lock callerFuncName is_dns_hijack;Dec  9 07:24:10 90.192.172.112 syslog: ssk:363.225:is_dns_hijack:1425:isDnsHijack=0Dec  9 07:24:17 94.15.218.140 syslog: ssk:563.048:cmsLck_acquireLockWithTimeoutTraced:95:acquired lock. callerFuncName is_dns_hijack; timeout 12000 millisecondsDec  9 07:24:17 94.15.218.140 syslog: ssk:563.048:cmsLck_releaseLockTraced:141:lock hold time=0ms, acquiring lock callerFuncName is_dns_hijack; releasing lock callerFuncName is_dns_hijack;Dec  9 07:24:17 94.15.218.140 syslog: ssk:563.049:is_dns_hijack:1451:isDnsHijack=0to this :Dec  9 07:24:02 94.15.218.140 syslog: ssk:548.049:is_dns_hijack:1451:isDnsHijack=0Dec  9 07:24:10 90.192.172.112 syslog: ssk:363.217:cmsLck_acquireLockWithTimeoutTraced:98:acquired lock. callerFuncName is_dns_hijack; timeout 12000 millisecondsDec  9 07:24:10 90.192.172.112 syslog: ssk:363.218:cmsLck_releaseLockTraced:144:lock hold time=0ms, acquiring lock callerFuncName is_dns_hijack; releasing lock callerFuncName is_dns_hijack;Basically, it is the same log repeated more times due to a different timestamp. I was trying to use uniq but in order to do that I need to remove the time stamp in the third column (achievable with  awk '{ $3=; print }') but as you can see from the log, the first 11 characters are different (i.e ssk:563.048 - ssk:563.049). I was thinking of doing a grep with the word dns and trying to ignore the first 11 values. How can I do this? Is there a better way?
GREP exclude first 11 values
bash;text processing;awk;grep
Use awk and : as the field delimiter. Then, you can save each unique error message (which will be the 6th field) in an array and only print the first occurrence:$ awk -F: '!a[$6]++' file Dec  9 07:24:02 94.15.218.140 syslog: ssk:548.049:is_dns_hijack:1451:isDnsHijack=0Dec  9 07:24:10 90.192.172.112 syslog: ssk:363.218:cmsLck_releaseLockTraced:144:lock hold time=0ms, acquiring lock callerFuncName is_dns_hijack; releasing lock callerFuncName is_dns_hijack;Dec  9 07:24:10 90.192.172.112 syslog: ssk:363.217:cmsLck_acquireLockWithTimeoutTraced:98:acquired lock. callerFuncName is_dns_hijack; timeout 12000 millisecondsThe script above will save each line as an entry in the associative array a only if there isn't already a value saved for the 6th field (!a[$6]). Because the default action in awk is to print when something evaluates to true, that will result in only the first unique occurrences being printed. If you only want to do this for lines matching dns, use:awk -F: '!a[$6]++ && /dns/' file As for ignoring the 1st 11 lines, you could do that with:grep dns file | tail -n +12