# Extract identical files from two directories

I am looking for a linux solution for the following problem:

Given two directories with a large number of files. All file names are random and different in both directories. The contents of some of the files in the two directories are identical however.

I want to copy all files which occur in both directories to a third directories. ("Occur in both" means have the same content not the same name.)

Assuming your filenames don't have any whitespace and there are no subdirectories in either directory, the following will print pairs of filenames with matching MD5 sums:

``````join -o 1.2,2.2 <(md5sum \$D1/* | sort) <(md5sum \$D2/* | sort)
``````

To get just one of the filenames, use `-o 1.2` or `-o 2.2`.

If filenames (or paths) might include whitespace, you'll need to be more clever.

If a single directory might have the same file with more than one name, you will also need to be more clever -- and you will need to decide exactly what to do. One possibility would be to filter out the duplicates before doing the join:

``````join -o 1.2,2.2 <(md5sum \$D1/* | sort | uniq -w16) \
<(md5sum \$D2/* | sort | uniq -w16)
``````

DO NOT USE `sum`

`sum` outputs a 16-bit checksum; if you have even a couple of hundred files in each directory, it is likely that you will get a false positive if you compare 16-bit checksums. `md5sum` is not absolutely definitively safe, either, but the odds of a collision with 128-bit checksums is tiny. In case of doubt, and if it's really important, `cmp` the files as well:

``````join -o 1.2,2.2 <(md5sum \$D1/* | sort) <(md5sum \$D2/* | sort) |
if cmp -s \$F1 \$F2; then
cp F1 \$D3
fi
done
``````

(Again, that won't work if the files might have whitespace in their names.)

Using your favorite shell for this pseudo code:

``````cd D1; sum * | while read l; do echo "D1 \$l"; done >/tmp/foo
cd D2; sum * | while read l; do echo "D2 \$1"; done >>/tmp/foo

sort -n /tmp/foo | awk '
\$1 == prev_cs { echo "cp \$3 dest"}
/prev_cs = \$1/
' | shell
``````

You can save the awk output for review before issuing the copies, if you like

• `sum` is really a bad idea, see en.wikipedia.org/wiki/Birthday_problem. If you have 300 different random files, the odds are roughly even that two of them will have the same `sum`. – rici Mar 15 '14 at 2:25
• Then pick a different checksum algorithm, like `openssl dgst -md5`. Same code. – mpez0 Mar 15 '14 at 12:05

this might do the job, as with mpez0's ans for copying the dups.

`find {tst1,tst2} -exec sum {} {} \; 2> /dev/null | sort | uniq `
{"ft_lang_label":"__label__en","ft_lang_prob":0.8710104,"math_prob":0.5847525,"size":1270,"snap":"2019-51-2020-05","text_gpt3_token_len":383,"char_repetition_ratio":0.123222746,"word_repetition_ratio":0.07725322,"special_character_ratio":0.30393702,"punctuation_ratio":0.121323526,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.95446074,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2019-12-09T10:31:49Z\",\"WARC-Record-ID\":\"<urn:uuid:f88b5503-ceea-4651-b088-e3a3cde3cf62>\",\"Content-Length\":\"143663\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:f4776941-b10f-4851-ab99-c06bdeb165e4>\",\"WARC-Concurrent-To\":\"<urn:uuid:4b090d6b-85b5-48cd-9d51-32480a1c9b4b>\",\"WARC-IP-Address\":\"151.101.193.69\",\"WARC-Target-URI\":\"https://superuser.com/questions/729226/extract-identical-files-from-two-directories/729238\",\"WARC-Payload-Digest\":\"sha1:KM2YVCHWCVTCLLVRU5HVAJJT75JZO5GP\",\"WARC-Block-Digest\":\"sha1:FQIKGMGFQAQGGGBB5FJXETGTDPOJJUQZ\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2019/CC-MAIN-2019-51/CC-MAIN-2019-51_segments_1575540518627.72_warc_CC-MAIN-20191209093227-20191209121227-00424.warc.gz\"}"}