Conflict of interests and the assignment of referee for a very narrow field
Even if the field is small, one can usually find referees in cognate fields with no conflicts: presumably papers by these groups have a bibliography that extends beyond these groups, else the interest is overly narrow and the paper can be rejected as such.

(Aside: the handling editor is not necessarily a “he”.)