How can I efficiently dedupe a very large number of records?
We did a lot of work to the dedupe processes in CiviCRM and in 4.7 its a much easier process to de-dupe large databases then it used to be. It was presented by Eileen in this years London CiviCon, worth a look before you get into trying to do it outside of CiviCRM with all the pitfalls that brings with it!