I recently learned of a program called Total Commander. It's a Windows Explorer replacement and has its own stuff to copy files. To check whether the files are identical, instead of calculation a CRC, it literally checks every single byte, one at a time, on both the original and the copy.My question is: Is this necessary? Can CRC or any other such technique go wrong? Should you, as a programmer, try and implement this perfect but slow system, or is it too extreme?
Is it necessary to read every single byte to check if a copied file is identical to the original?
difference;file handling
Calculating CRCs (or, better, sha1sums) on both files requires reading every byte anyway. If you do a byte-by-byte comparison, you can quit as soon as you see a mismatch -- and you don't have to worry about two different files that happen to have the same checksum (though that's vanishingly unlikely for sha1sum).  So if you're doing the comparison locally, a byte-by-byte comparison will be at least as fast as a checksum comparison (unless you've already computed the checksums anyway).On the other hand, checksum comparisons are useful when you're comparing files that aren't on the same machine; the checksums can be computed locally, and you don't have to transfer the entire content over the network.Hybrid approaches are also possible. For example, you might compute and compare checksums for the two files a chunk at a time, which can avoid reading the whole files (if they differ) while also avoiding transmitting the whole file across the network. The rsync protocol does something like this.Note that using a simple CRC gives you a fair chance of a collision, as Dave Rager mentioned in his answer.  Use at least sha1sum, or even something more recent.  (Don't try to invent your own hashing algorithm; the folks who developed sha1sum know far more about this stuff than either of us.)As for the likelihood of collision, if you use a decent hash like sha1sum you pretty much don't have to worry about it, unless someone is deliberately and expensively constructing files whose sha1sums collide (generating such collisions was not feasible when I first wrote this, but progress is being made).  Quoting Scott Chacon's Pro Git, section 6.1:Heres an example to give you an idea of what it would take to get a  SHA-1 collision. If all 6.5 billion humans on Earth were programming,  and every second, each one was producing code that was the equivalent  of the entire Linux kernel history (1 million Git objects) and pushing  it into one enormous Git repository, it would take 5 years until that  repository contained enough objects to have a 50% probability of a  single SHA-1 object collision. A higher probability exists that every  member of your programming team will be attacked and killed by wolves  in unrelated incidents on the same night.Summary :Byte-by-byte comparison is good for local comparisons.  sha1sum is good for remote comparison, and presents no significant chance of false positives.