Best approach for a random terrain rolling ball animation?
I would manage all your keyframes as you have been doing, set the ball to be a Rigid Object with Dynamic property, and the terrain to be a Rigid modifier, too - but the collision type set to mesh, and then bake it.
Once you've done that, you could then convert the bake to keyframes, and tweak it to your liking.