Why does an affine transform work only on three of the corners?
An affine transformation doesn't have enough freedom to do what you want. Affine transforms can be constructed to map any triangle to any other triangle, but they can't map any quadrilateral to any other quadrilateral.

One way to see this is that the matrix for a 2D affine transform has only 6 free coefficients. That's enough to specify what it does to 3 points, but not 4.

For this task you likely want a projective transformation. A 2D projective transformation has enough freedom to map any convex quadrilateral to any other.