Should we use all $N$ correspondence points to compute Homography or just $4$ points?
It's generally better to use all the points you have, since often, errors are independent per point.

given my correspondence points are good, shouldn't there be an exact
solution

Sure, but in realistic situations, it's not usually the case and/or you don't know that the points are good / which points are good.
A typical approach is to use all the points you have, but reject &quot;bad&quot; points using outlier-rejectiong  (e.g. RANSAC ) These slides have an example of such a procedure.