Animating a smooth linear transformation
As a general rule, you cannot interpolate transformation matrices. In stead, you decompose them into their individual values, then interpolate those and recompose. 

The Möbius transformation as suggested in the comments sounds interesting, but tradionally I'd just extract scale and rotation and interpolate those.

Assuming a transformation matrix

    | a  b |
T = | c  d |



the scale is [sqrt(a² + c²), sqrt(b² + d²)]
The rotation angle is atan(c/a)


You'd now interpolate this scale and rotation angle, recomposing the matrix at each frame.

[Edit] Corrected the angle (as spotted by Nathan Reed) [/Edit]