Why SqlBulkCopy gets slower by table's row count?
Why you write that you don't make any index ?
What is PK_DeliveryReports in your opinion ? 
As for me - the clu of your problem is just rebuilding of PK.
Try to use partitions to your table - or drop this index (PK) and rebuild it after the load - should be quicker (of course if you don't need index during the load for other purposes).