Is there any software for data management?
While it doesn't cover all your criteria, I use a program called Total Commander for most of my file management needs and it works well for working with a very large number of files and huge file sizes.

Among it's features, it is able to do a directory synchronization that determines the differences between to folders, either by filesize/date or by content (though a content comparison across a network share is going to take a very long time).  

It doesn't have the queue or resume functionality you want, but it handles duplicates well and is pretty easy to work with overall.  

It does include a pause feature on the copies and lets you run multiple copies at the same time in the background.

It does include a variety of powerful file selection tools to let you choose particular sets of files that need to be copied.

It has a good UI and is relatively cheap.  It is available as shareware to try for free and last I knew it was $35 for a lifetime license.  My dad bought a copy when Windows 3.0 was still the current OS and hasn't paid again since.  I got my license in 2002 or so.