Are men subject to optimism bias when it comes to assessing their sexual/romantic appeal? Intuitively, I imagine that men are strongly subject to optimism bias when dating, and in particular facing rejection. (ie. not believing that she's really not attracted to him). 

My question is:


How well studied is this? What methods of study do they use?
How prevalent is it amongst men? (Are all men subject it it?)
How strong is the bias? By how much do men overvalue their attractiveness? 
How does this compare with women?
Short answer: Yes, but not really...

Self-enhancement:

Self-enhancement (sometimes referred to as positive illusions) refers to a general preference for positive self-views (in men and women alike).  It includes several common strategies, such as: The "above average effect" (aka illusory superiority), self-serving bias, and optimism bias.  Optimism bias however refers to perceived risk, whereas this question is about illusory superiority:


  ... a cognitive bias whereby individuals overestimate their own
  qualities and abilities, relative to others. This is evident in a
  variety of areas including intelligence, performance on tasks or
  tests, and the possession of desirable characteristics or personality
  traits.


Self-esteem is a particularly dicey subject matter - it seems to refer to a conglomerate of fairly independent / mildly correlated factors.  So when we examine say, self-perceived sexual attractiveness (sex appeal), the results are different from a variety of other similar sounding - but independent - constructs such as self-perceived physical attractiveness, sexual self-schema, self-valuation, sexual self-esteem, sexual satisfaction, sexual performance, body image, self-image, etc...

Therefore, to keep the scope of this question manageable, I'll focus on gender differences in illusory superiority with respect to sexual attractiveness, though as will be discussed, physical attractiveness will come into play as well.

Research methodology:

Self-esteem factors such as perceived sexual attractiveness are measured using surveys / self-reporting.  For sexual attractiveness in particular, there doesn't appear to be a well-established scale in common use, so researchers tend to develop questionnaires as needed for their particular studies.  Furthermore, there is evidence that many confounding factors exist that cannot be easily controlled for, such as societal changes.  So unfortunately, some questions of validity and reproducibility of the results exist, as is common in many facets of social psychology and personality research.

Another issue is the extricability of sexual attractiveness from physical attractiveness.  Studies have shown that these are in fact two different constructs.  However, different scales used by different researchers emphasize physical attractiveness in sexual attractiveness measures to different degrees, making the separation difficult.

As suggested in the question, self-enhancement is known in general to increase when under threat - Wikipedia says:


  This motive becomes especially prominent in situations of threat,
  failure or blows to one's self-esteem.


and:


  Positive illusions are a form of self-deception or self-enhancement
  that feel good, maintain self-esteem or stave off discomfort at least
  in the short term.


To manipulate self-esteem in the lab, subjects can be presented with a comparison point (contrast), such as a photo or video of someone else that may be highly attractive, or unattractive.  An ego-threatening contrast may affect self-enhancement differently from an unthreatening scenario.

Summary of results:

As expected, both men and women report a "better-than-average" bias for sexual attractiveness - ie, the average rating compared to others is higher than average.  The bias is slightly greater for men than women on some scales, and not significantly different on others, eg:


  ... on the dimensions of Sexual Attractiveness, Sexual Restraint, and
  Sexual Orientation ... no significant sex differences [were] found.


Here is an example where women actually rated themselves higher than men:


  ... we found that males rate themselves higher on explicitly sexual
  dimensions (e.g., sexually responsive, experienced) and females
  perceive themselves to be more romantic and sexually attractive.


It would be good to see a meta-analysis of such studies to get a better idea of the overall effect, but the bottom line is that it's not clear - it may depend on the particular scale used, or any number of other confounding factors.

Some examples of such factors: Black men rate themselves higher than white men, but women do not; men and women base their ratings on physical traits to different degrees; self-esteem affects females differently from males; and effects vary in cultures where physical attractiveness is more or less emphasized.  That is, it appears that gender differences vary based on societal changes, such as before and after the feminist movement, differences in idealized representations in the media, cultural changes in messages such as "black is beautiful", etc.  And gender differences also vary depending on the incorporation of physical attractiveness.

Physical attractiveness:

In the 1980's and 90's, there was a flurry of research on self-perceived body image and physical attractiveness related to increased diagnosis of anorexia/bulemia, so much more is known about these measures.  In line with suggested epidemiology for anorexia, that is most associated with young women, body image is typically lower in females than males, and females are more often subject to a "lower-than-average" effect, while males more often rate their physical attractiveness as better than average.

As before, there are many confounding factors, including some of the same ones to do with skin colour, cultural context, different physical traits, and self-esteem.  Nonetheless, the effect suggested in the question is perhaps more pronounced for physical attractiveness than sexual attractiveness / romantic appeal.  And since self-perceived physical attractiveness often constitutes an important component of sexual attractiveness, then some of the differences in results may depend on how heavily a particular study incorporates physical attractiveness into their own measure of sexual attractiveness, and how important physical attractiveness is as a component of sexual attractiveness for the particular group studied.  With this factor removed, and limiting findings to modern Western culture, it may turn out that men and women do not differ in self-enhancement of sex appeal, or even that women do so to a greater degree.