term for species that produces males only to mate with mother or sisters
The non-random mating of individuals within a population is called Assortative Mating. Mating specifically with individuals that share similar traits is called Positive Assortative Mating. However, sharing similar traits doesn't imply being closely related.

The reproduction by two closely related individuals is called inbreeding. The opposite of inbreeding is outbreeding (a.k.a. outcrossing).

To my knowledge, a mating system where all individuals inbreed doesn't have a special term. Searching google.scholar for mating system inbreeding will show you a lot of articles that talk about mating systems where all individuals inbreed without needing to use a specific term for it.