Matrix from quaternion produces a different orientation
Your quaternion is not normalized, this can happen during interpolation.

Blender will still only view the quaternion as an orientation, even if it is not pure. However, the to_matrix function will convert the quaternion as it is.

To circumvent this, simply normalize the quaternion before converting it to a matrix.

ob.matrix_world = ob.rotation_quaternion.normalized().to_matrix().to_4x4()