Copying keyframes makes objects snap to one point
Inorder to prevent fractured pieces from going into a single point, move all the fracture origins to the center of the ball. You can do this by selecting all the fractures, Shift+S, Cursor to selected, Ctrl+Atl+Shift+C, Origin to 3D cursor. And then copy the animation data to other pieces or do what I did in the below file.

Please check the layer2 in attached files to see how I implemented the collision.


I parented the fractures to empties and animated the empties.

For all the fractures just until a few frames before the collision (at about 70 frame), I kept rigid body 'dynamic' option off and 'Animated' on. After this point, 'Animated' off and 'dynamic' on. Let me know if you need more clarification.