Filtering RANSAC Estimated Homographies
There is a problem in checking whether the homography is OK.

The algorithm for checking correct homographies may interest someone, so I will write it down here:

1) Create a quadrilateral $ABDC$ with vertex coordinates (in homogenous coordinates):

$$\begin{eqnarray} A:&amp; (-w/2,-h/2, 1.0) \\ B:&amp; (w/2,-h/2, 1.0) \\ C:&amp; (-w/2,h/2, 1.0) \\ D: &amp;(w/2,h/2, 1.0) \end{eqnarray}$$

where $w,h$ are width and height of the image, respectively. If the whole image frame (a rectangle) is transformed to convex quadrilateral, than any convex quadrilateral within it will also be transformed "neatly".

2) Create transformed quadrilateral $A'B'D'C'$ in which every vertex is transformed using the computed homgraphy (e.g. $C'=HC$ ). From now on, all points will be converted to non-homogenous coordinates.

3) Compute vectors $\vec{u}$, $\vec{v}$ for parametric representation of diagonals:

$$\begin{eqnarray}d_{1} :&amp; A+(D-A)s =A+ \vec{u}s \\ d_{2} :&amp; B + (C-B)t=B+\vec{v}t \end{eqnarray}$$

The intersection of diagonal comes from $d_{1}=d_{2}$:

$$t=\frac{1}{d}\left[(B_{y}-A_{y})\vec{u}_{x} - (B_{x}-A_{x})\vec{u}_{y}\right]$$

$$s=\frac{1}{d}\left[(A_{x}-B_{x})\vec{v}_{y} - (A_{y}-B_{y})\vec{v}_{x}\right]$$

Then the convex quadrilateral satisfies $s,t \in (0,1)$.

In practice, one can introduce a fudge factor to avoid not only non-convex and degenerate quadrilaterals, but also near-degenerate ones, like when three points are near colinear. So the test can be modified such that $s,t \in (\lambda, 1.0-\lambda)$, where lambda is a small number (e.g. $\lambda=0.01$).

Older problem, fixed in the above algorith:

I found the problem here - having a certain homography, the test can pass for a smaller quadrilateral, but not for the larger one. This is why some "ill" homographies passed through.

The green square represents a source image, the orange is a transformed one. As you can see, the left hand one is convex, but starts deforming as the source is larger:



Finally even larger source yield non conver quadrilateral:



I made a mistake with scaling. The points in homogenous coordinates $(x,y,w)$ were scaled in $x$ and $y$ direction, but in $w$. This is why the same transform produced different quadrilaterals.

I have corrected the algorithm accordingly.