How can I find duplicate videos from a large set of videos of varying formats?
I'd recommend a hybrid approach using both computers and people.


Bucket the videos by their length (round to the nearest second)
For each bucket, use ffmpeg to generate thumbnails at a predictable and uniform point in the videos (ex: a frame from 10 seconds into the video)
Look at the generated thumbnails in a grid (most OS's provide a nice thumbnail view) and scan for duplicates to remove.


You shouldn't have to do any programming to perform these steps, though creating the thumbnails on the command line with ffmpeg may take some finesse.  Good luck!