How does temporal reprojection work?
One strategy mentioned in Brian Karis's talk about TAA is neighborhood clamping. The general idea is that, for the previous frame's pixel to be valid, its color should be in the color range found in the neighborhood (say 3x3 pixels) of the current pixel this frame.

This rejects history from changing light conditions, which is probably what you want anyway if you don't want moving shadows to produce ghosting.

(Animated textures, depending on the speed of the animation, could also be handled with a motion vector, if you have a predictable UV mapping or can guess reasonably well.)