I've searched around the internet and stackexchange for this. Even though there are lots of similar topics, I haven't found a solution yet. So, I have a quite large list (approx. 20GB), which contains around 5% duplicate lines. I want to filter this list, sothat one of the duplicates is deleted. Example:Input:test123Test123testtest123test 123Output:test123Test123testtest 123Whether the list gets sorted or not doesn't matter.I've tried sort -u -o output.txt, also sort -us -o output.txt. It works fine for smaller files, but when I try to do files of more than approx. 4GB, the resulting file is suspiciously small and, instead of a .txt file, it has apparently become an emacs-lisp-source-text.I'd be very grateful if someone could help me out!
Removing duplicates in a large text list
linux;shell;text processing;files;sort