Using Cell Fracture the object that is supposed to break, explodes or breaks before impact
The Cell Fracture and physics process by default start at frame 1, and that has to be delayed.  After you do the fracture, select all segments and press  Ctrl + L  to link them all. Choose Animation data.
Now what you keyframe on one will affect them all.  Select one segment, set and keyframe it's physics to Animated at frame 1. Untick and keyframe the Dynamic box also.
Move the timeline to the collision frame (or there-abouts) and keyframe the boxes in reverse.  Animated unticked and Dynamic ticked.
Addendum 1st April 2021

One thing further to the above.  If the physics are still activating at frame 1, save the file and exit Blender.  Reload it.  This will clear the physics cache.

Apparently the procedure has been changed in vers 2.8+ and the new method is set out here - 
linking objects in blender 2.8