How do I ask a professor to contact another professor he knows to accept me as a Masters student *via email*?
Since you aren't now his student and can't visit in person, I'd send a rather longish mail first requesting that he do you this favor, sending a letter to "Prof X", then giving the background that you've been unsuccessful and finally reminding the prof of the old work you did and any relevant updates. 

I think that making the request first will permit him to quit reading at any point so that the length of the mail is less important as normally a short mail is appreciated. 

"Longish" implies about a page with about 3 paragraphs, not a monograph. 

Somewhere along the way note that you really want to work with his relative and that you know of the relationship. But don't make that the main focus. Letter from Prof A to Prof X. 

But I don't think you are asking a lot. Part of his job is to send out such letters. It may be that sending it to a relative will cause him to send a different sort of letter, but perhaps not. The other prof may take a recommendation from a relative as having more weight (honesty) or not. But that is up to them.