SSIS data flow task failed for no reason
Think about how the reference and input data sets are configured.
The reference data set (smallest set) should be the only cached data set, while the input rows(i.e. the 10M) pass through that execution tree's buffer.

Also, running parallel lookups on different subsets of the data and aggregating all outputs with a UNION ALL at the end might yield better performance, but test it.

Lasty, look for any block transformations (i.e. Sort, Aggregates, etc.) before or after the lookup.

Getting Optimal Performance with Integration Services Lookups

Integration Services: Performance Tuning Techniques