Ray Transformation to Object Space for Motion Blur
Lerping the ray positions/directions between keyframes should be equivalent to lerping the inverse matrices between keyframes and transforming by the lerped matrix. Trouble is, if the keyframes have different rotations, that lerped matrix will in general be something "weird", with shearing, nonuniform scale, etc.

I wouldn't be surprised if some of your intersection and shading routines don't work properly in such a distorted coordinate system, unless you've specifically tested and hardened them against such cases. (For example, taking the dot product of two unit vectors doesn't give the same answer in a sheared coordinate system as in an orthonormal one.)

This is just a guess, but it might work better if you choose an interpolation method where the translation, rotation, and scale (if applicable) are lerped separately (using quaternions for the rotation part) and re-combined.