How do I get a Falloff offset matrix to look like it's coming from a single point?
Subtracting the initial position of the objects from the offset vector should do what you want, it is also non-destructive, so it might be better than your original solution: