Affine transformation without imwarp I program function to calculate the overlap area between 2 images (which is called fixed and floating) after transformation but it takes much time and it's complex for me to combine more than one transformation parameter together.

In Matlab, the built in functions affine2d can make affine transformation (rotation, translation, and scaling) that can solve my problem. However, this function depends on imwarp which fills the empty space with new pixel (default: black).

As you can see the below image, my ROI inside the red box in the both images.



How can extract this area from the transformed image (float)? 

Update: the hardest part is to find the ROI coordinator in the transformed image.
To find coordinate or index of drew region, first you have to find the coordinate of 4 colored dots, which is easily obtained by using the transformPointsForward function over the intrinsic coordinate of these 4 points in your original image. intrinsic coordinate is the index of your image matrix.

then you have to do some geometric calculations. first, find the length of one of red lines which are the (length of full warped image)/(sin(theta) + cos(theta)), then distance of red line to the nearest colored point is (length of red line)*cos(theta)*sin(theta). theta is the rotation angle. using this distance and the coordinate of 4 colored points you could find the coordinate of your ROI corners.