PostgreSQL "upsert" tuning with unique constraint
Partitioning the table would only help if the data in the files you are trying to insert are also partitioned using the same scheme.

If you sort the data to be inserted into the same order as the unique index is defined on, that should help if the bottleneck is reading from the index.  But if the table is not clustered in the same order as the index, and the bottleneck is reading from the table, this might not help much.

For this type of task, I would usually deduplicate outside the database.  You might also look into using something like file_fdw to map the new data into the database, and then query and insert for new data.