What is the "male gender rudiment" that women allegedly have?
First, let's briefly discuss the development of the the male and female internal genitalia and sex glands:

These include the ejaculatory duct, prostate and seminal vesicles for males, and uterus, uterine tubes and the proximal part of the vagina for females. 

They all develop from two tube-like structures called mesonephric and paramsonephric ducts.

These ducts initially exist in both males and females, but normally, only one develops while the other degenerates, depending on the presence of sex-determining factors and hormonal activity.

The mesonephric ducts will develop to the male internal genitalia (the organs are mentioned above), while the paramesonephric duct will develop to the female internal genitalia (also mentioned above).

Now, to your question:

A male rudiment that exists in females is the non-fully degenerated mesonephric duct, named the Epoophoron and Paroophoron in adults, which later in life may form the Gartner’s cyst.