How to warp a pixel patch to another camera
$x  K_{inv}$, with $x(3) = 1$, produces the pixels as 3D points "stuck on the screen" of the first camera. After transforming you get the same 3D points, but in the coordinate frame of the second camera, and then you project in that camera. So, overall, what you are doing is looking at the first camera from the point of view of the second one.

I suspect that is not what you want....