it was known in a very close circle , some members of the royal family and senior bishops , that edward iv was not a legitimate heir to the throne but the product of an adulterous affair between the wife of richard , duke of york , and a common archer in the dukes army .