Dynamic Paint; invisible object is acting as brush anyhow
There are a few ways to do this, but both require new properties to be animated in sync with when you are animating the object's visibility.

The first way is to animate the visibility options on the Dynamic Paint modifier (on the brush object):



You can do this either by right-clicking on the icon and choosing 'Insert keyframe' or by pressing I when the cursor is over the icon. Note: You have to keyframe both the 'Show Viewport' (screen icon) and 'Show Render' (camera icon) settings to get this to work. Animate these values to be on when the object is visible and off when it isn't.

The second method is to animate the 'Alpha' value on the brush object to be '0' when you want it to have no effect and '1' when you do:



You may also have to animate the 'Wetness' value, depending on your setup.

Note that you should be able to copy your existing keyframes onto these properties by using the 'Dope Sheet' editor (but you must add at least one keyframe to these properties before they will appear in the Dope Sheet) by using the Key> Copy Keyframes and Key> Paste Keyframes menu options.