How do I get a Falloff offset matrix to look like it's coming from a single point? In my example file the LEGO car looks like it's being assembled from a central point and it's okay when it's relatively close in size to the icosphere I am using as a prop. 



Relatively close in size



You can see here the objects are in the models final structure



It does not look so good when the model being assembled is much larger than the icosphere prop. The pieces come from a much wider range.

Relatively large



I've tried to adjust initial transforms but then I loose my desired built model. 

My question is how can I maintain the same effect of the model being assembled and the pieces are coming from a single point?
Subtracting the initial position of the objects from the offset vector should do what you want, it is also non-destructive, so it might be better than your original solution: