Blender Fracture Modifier locking to another mesh
You can add a fracture modifier to the handle too, setting shard count to 1 if you dont want it shattered. But it is important it is an FM object in order to use "Automation" -> "Combine" -> "Connect" in the fracture helper addon which is shipped with the fracture modifier build. (Needs FM 2.79a or newer.)

This will connect FM objects with external constraints where the objects "touch" basically. You can control the properties of the external constraints on the newly generated helper object (which has also an FM, but only maintains the constraints).

Alternatively you can setup the connection manually. In order to do so you can put the desired FM objects into a group, then create another helper object manually. Put an FM onto it, disable all sources. Specify the Group in "Sub object group" and check "Constraints only"... also enable "Use constraints in the helper object. But note, its easier to let the addon do the boring automation work.

Here I quickly built an example file with a simple door :)