Is there a way to keyframe a modifier to turn on or off?
To achieve this effect you can use the Dynamic Fracture Mode for the object to be fractured. Note that the Dynamic threshold will cause the object to fracture on impact after falling if set too low. Make adjustments between this and the mass of other objects that interact with the simulation so that things work as desired. 



The object that will fall onto this object needs to have a sufficient amount of mass in order for the fracture to work. Here, the sphere has a mass of 25, the rest of the rigid body settings are at defaults. 

You may need to increase steps per second in the scene properties if the simulation seems to be unstable, just be sure that all transforms (rotation, scale) are applied.