Intuitively, I imagine that men are strongly subject to optimism bias when dating, and in particular facing rejection. (ie. not believing that she's really not attracted to him).
My question is:
How well studied is this? What methods of study do they use?
How prevalent is it amongst men? (Are all men subject it it?)
How strong is the bias? By how much do men overvalue their attractiveness?
How does this compare with women?
Short answer: Yes, but not really...
Self-enhancement:
Self-enhancement (sometimes referred to as positive illusions) refers to a general preference for positive self-views (in men and women alike). It includes several common strategies, such as: The "above average effect" (aka illusory superiority), self-serving bias, and optimism bias. Optimism bias however refers to perceived risk, whereas this question is about illusory superiority:
... a cognitive bias whereby individuals overestimate their own qualities and abilities, relative to others. This is evident in a variety of areas including intelligence, performance on tasks or tests, and the possession of desirable characteristics or personality traits.
Self-esteem is a particularly dicey subject matter - it seems to refer to a conglomerate of fairly independent / mildly correlated factors. So when we examine say, self-perceived sexual attractiveness (sex appeal), the results are different from a variety of other similar sounding - but independent - constructs such as self-perceived physical attractiveness, sexual self-schema, self-valuation, sexual self-esteem, sexual satisfaction, sexual performance, body image, self-image, etc...
Therefore, to keep the scope of this question manageable, I'll focus on gender differences in illusory superiority with respect to sexual attractiveness, though as will be discussed, physical attractiveness will come into play as well.
Research methodology:
Self-esteem factors such as perceived sexual attractiveness are measured using surveys / self-reporting. For sexual attractiveness in particular, there doesn't appear to be a well-established scale in common use, so researchers tend to develop questionnaires as needed for their particular studies. Furthermore, there is evidence that many confounding factors exist that cannot be easily controlled for, such as societal changes. So unfortunately, some questions of validity and reproducibility of the results exist, as is common in many facets of social psychology and personality research.
Another issue is the extricability of sexual attractiveness from physical attractiveness. Studies have shown that these are in fact two different constructs. However, different scales used by different researchers emphasize physical attractiveness in sexual attractiveness measures to different degrees, making the separation difficult.
As suggested in the question, self-enhancement is known in general to increase when under threat - Wikipedia says:
This motive becomes especially prominent in situations of threat, failure or blows to one's self-esteem.
and:
Positive illusions are a form of self-deception or self-enhancement that feel good, maintain self-esteem or stave off discomfort at least in the short term.
To manipulate self-esteem in the lab, subjects can be presented with a comparison point (contrast), such as a photo or video of someone else that may be highly attractive, or unattractive. An ego-threatening contrast may affect self-enhancement differently from an unthreatening scenario.
Summary of results:
As expected, both men and women report a "better-than-average" bias for sexual attractiveness - ie, the average rating compared to others is higher than average. The bias is slightly greater for men than women on some scales, and not significantly different on others, eg:
... on the dimensions of Sexual Attractiveness, Sexual Restraint, and Sexual Orientation ... no significant sex differences [were] found.
Here is an example where women actually rated themselves higher than men:
... we found that males rate themselves higher on explicitly sexual dimensions (e.g., sexually responsive, experienced) and females perceive themselves to be more romantic and sexually attractive.
It would be good to see a meta-analysis of such studies to get a better idea of the overall effect, but the bottom line is that it's not clear - it may depend on the particular scale used, or any number of other confounding factors.
Some examples of such factors: Black men rate themselves higher than white men, but women do not; men and women base their ratings on physical traits to different degrees; self-esteem affects females differently from males; and effects vary in cultures where physical attractiveness is more or less emphasized. That is, it appears that gender differences vary based on societal changes, such as before and after the feminist movement, differences in idealized representations in the media, cultural changes in messages such as "black is beautiful", etc. And gender differences also vary depending on the incorporation of physical attractiveness.
Physical attractiveness:
In the 1980's and 90's, there was a flurry of research on self-perceived body image and physical attractiveness related to increased diagnosis of anorexia/bulemia, so much more is known about these measures. In line with suggested epidemiology for anorexia, that is most associated with young women, body image is typically lower in females than males, and females are more often subject to a "lower-than-average" effect, while males more often rate their physical attractiveness as better than average.
As before, there are many confounding factors, including some of the same ones to do with skin colour, cultural context, different physical traits, and self-esteem. Nonetheless, the effect suggested in the question is perhaps more pronounced for physical attractiveness than sexual attractiveness / romantic appeal. And since self-perceived physical attractiveness often constitutes an important component of sexual attractiveness, then some of the differences in results may depend on how heavily a particular study incorporates physical attractiveness into their own measure of sexual attractiveness, and how important physical attractiveness is as a component of sexual attractiveness for the particular group studied. With this factor removed, and limiting findings to modern Western culture, it may turn out that men and women do not differ in self-enhancement of sex appeal, or even that women do so to a greater degree.
Respectfully, I believe there might be a flaw in the thinking behind the question insofar as you are assuming that "attractiveness" is a relatively static property in an individual.
This might be a male perspective because the way that males are programmed to feel attraction for females is based on female attributes which are relatively slow to change.
Females, however, are programmed very differently than males. The amount of attraction or repulsion that she feels can change significantly very quickly because it is largely based on the behavior of the male. This is a concept that most males find hard to grasp because it is not the way that they are programmed so it is not a part of their experience of reality.
An explanation ...
The field of evolutionary psychology suggests that much of animal behavior - including and especially our attraction and repulsion systems is hardwired into our genes.
For example, when a cat hears the squeaking of a mouse, the behavior of the the cat will be somewhat predictable. How is this possible? The belief is that cat has been genetically programmed by evolution to feel an emotional compulsion to behave in a very specific way when it receives a very specific auditory signal. This behavior gives it an evolutionary advantage over those cats which did not have it because it will eat the mouse and the nourishment will give it a better chance of surviving to pass on its genes.
Evolutionary psychology suggests that much of animal behavior - including and especially mate selection choice - is similarly hard-wired. Those who make good choices for selection of mates are more likely to have their genes survive.
However, the programmed mating strategy of males and females is very different for the simple reason that the optimal strategy for the survival of the genes of the male is different than the optimal strategy for the survival of the genes of the female.
In particular the male mating strategy might be described as seeking high “replication value” whereas the female strategy is to seek high “survival value”.
Men have the advantage of being able to make quick and relatively permanent assessments of female attractiveness because all they need is a visual signal to assess her “replication value” and that assessment - before the advent of plastic surgery - is relatively permanent. Females, however, cannot assess the “survival value” of a prospective mate so quickly. She has to get to know you in order to make an accurate assessment. The level of attraction that she feels for you will be based nearly entirely on how you behave during that “getting to know you” time.
A popular book on the evolutionary psychology of attraction is "The Evolution of Desire: Strategies of Human Mating" by David M. Buss.
Evidence to support the belief that the attraction mechanism of females is rooted in male behavior is given by the (admittedly non-scientific) experience of “pick-up artists”. For example “Eric Von Markovic” aka “Mystery” wrote a famous book called “The Mystery Method” in which he describes how a male can alter his behavior to “create attraction” in women by behaving in ways that exude the desirable traits and not behaving in ways that exhibit the undesirable traits.
I realize this is not a rigorous scientific source. Someone should do a study to validate the claims. But I personally have a great deal of certainty that the method works because I was a dating coach in a former life, taught it to dozens of men, and saw it work with my own eyes over and over again in the field.
This brings me to my second point.
One of the most important behavioral traits that women seem to be programmed to seek out in men might be described as “self confidence”.
So if you, as a male, have a genuine “optimism bias”, females will actually feel more attracted to you because you are more self-confident. Women are attracted to men who have this trait and feel aversion to those who don't have it.
I do not have a scientific reference to support this. However, all that I have experienced tells me that it is true. You might consider doing a simple survey of women to verify.
In his book, von Markovic suggests that women seem to be programmed to run a “testing” algorithm to ascertain if you really are the attractive mate who can fulfill her needs that you seem to be. If a male “behaves” correctly, her attraction will increase. If he behaves incorrectly, the female’s attraction will decrease.
So all men are punished and rewarded by their life experiences to develop the attractive behavioral traits - including and especially “self-confidence”. It would seem, therefore, that men are rewarded for having the trait of “optimism bias” and punished for not having this trait. Therefore a male who fails to find a way to generate self confidence with respect to his desirability is putting the survival of his genes in jeopardy by severely limiting his mating opportunities.
Are all men subject to it?
All men are compelled by their life experiences to develop it. Though many give unskillful meaning to and draw the wrong lessons from their painful experience and are thus much slower to develop the correct behaviors.
How is it different for women?
Men are not biologically programmed to punish women for lacking self confidence because we are not looking for her to be our "protector and provider". Women are therefore not compelled to develop this trait by their life experiences.
