Diff over large subdirectories fails
For a week I've tested a recommended solution I found at iFixit, when someone encountered an error code -50 on an external HD. It seems to resolve the issue I had. I also came across error code -50 before when I tried copying files across, but didn't immediately link this to my issue of failing diffs. I had the suspicion that some inactivity on the external hard drives was responsible for the failure and the article seems to confirm that. To quote from there for the solution should you run into similar problems:

To work around this error you'll want to go into System Preferences-&gt;Energy Saver and for both Battery and Power Adapter tick the box next to &quot;Put the hard disk(s) to sleep when possible&quot; on your computer. Yes that means the OS will put all of your drives to sleep when it can but that's the only way I've found to fix the problem.
What seems to happen is that some external drives have firmware which detects inactivity and spins the drives down. If OSX is not configured with the energy saver setting mentioned above then OSX is not expecting the drives to go to sleep. When accessing the drive after it puts itself to sleep obviously something gets messed up and the error -50 is thrown among other problems.
By configuring the OS to put disks to sleep the OS will issue spin-up commands.

The one thing that puzzles or surprises me though is why diff would report actual differences in files, when in fact it couldn't access(?) the file properly.