I've a program producing large amount of redundant information. I'd like to store only unique lines produced on stdout, but I can't do it in multiple steps, like:./prog > outcat out | sort | uniq > out2out file would be too big. How can I achieve a stateful uniq functionality, where the stateful_uniq would consume lines and emit only the ones it didn't see before?
Stateful uniq functionality
pipe;uniq
Doesn't ./prog | sort | uniq > out2 do what you want? Or even ./prog | sort -u > out2 if your sort supports it...To see unique lines as they come out without sorting the output, you can use an associative array with awk:./prog | awk '!(count[$0]++)'As pointed out by Stphane Chazelas, sort limits its memory use using temporary files; the awk solution doesn't, so if memory use is a concern then it might not be appropriate.