Firstly, this is my first post here and kudos to everyone here for making this happen. My linux skills are averageI collecting /var/log/messages files from several linux machines and dumping them to a remote box. On the remote box, i interrogate these files and filter them for couple of keywords (fail, error, fault etc), and this remote box mail them to me so that i can review them. That's all working OK, my problem is that, the filtered output file is of several thousand of lines. Most events are repeated, except timestamp that is different per line.I would like to have a script that will first filter the output file and truncate lines with certain strings, and also remove duplicated lines (remember the lines start with different timestamps :) ). I don't need some of the lines, such as 10:Aug  6 06:00:01 hostname crond[30256]: pam_sss(crond:account): Access denied for user oracle: 10 (User not known to the underlying authentication module)11:Aug  6 06:15:01 hostname crond[1637]: pam_sss(crond:account): Access denied for user oracle: 10 (User not known to the underlying authentication module)The legitimate lines would be things like hardware failure notification etc.I hope i'm clear enough, if not please let me know where i need to clarify further.Thanks a mil!
filter offline copy of /var/log/messages
bash;awk;sed;grep;cut