Fast data loading with duplicate check
There are many articles already on performance tuning of INSERT or `BULK INSERT' queries. Main points:


Optimize hardware: RAID/SSD/Fusion IO etc.
Tune source file system and network: copy files locally, use JUMBO packets if reading over network
Find optimal batch size, minimum logging
Insert into table with no indexes, triggers, constraints etc.
Apply all calculations, lookups, indexes, compression etc after load or in chunks
After all data is ready AND main table is stored on same disks, then switch partition - usually it will be almost instant as it will be a metadata only operation. However there cases when it is not.